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Abstract 

,  The  acquisition  of  expert  knowledge  is  recognized  as 
one  of  the  major  hurdles  facing  the  expert  system  programmer 
or  knowledge  engineer.  Unfortunately,  knowledge  acquisition 
is  seldom  addressed  in  any  detail  in  expert  system 
1 iterature,  even  though  there  exist  a  number  of  different 
techniques  that  a  knowledge  engineer  can  use  to  capture 
expert  knowledge.  The  purpose  of  this  study  was  to  identify 
and  evaluate  the  relative  ef feet i veness  of  three  knowledge 
acquisition  techniques  that  may  be  used  when  developing 
expert  systems  for  project  management  related  tasks.  The 
three  techniques  were  interview,  concept  mapping,  and 
interruption  analysis.  An  experiment  was  conducted  and 
quantitative  measures  of  effectiveness  were  derived.  ;  Jhese 
measures  included  technique  implementation  times,  times  to 
formulate  expert  system  production  rules,  the  total  number 
of  rules  and  the  inferences  required  to  complete  those 
rules,  the  number  of  translations  of  the  expert  knowledge  to 
encode  an  expert  system  rule,  and  the  contribution  each 
technique  made  to  a  final  expert  system  that  represented  an 
aggregate  of  all  the  expert  knowledge  collected.  While  no 
one  technique  was  clearly  superior,  used  together,  concept 
mapping  and  interruption  analysis  were  found  to  be  powerful 

tools  for  acquiring  expert  knowledge. 
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AN  EMPIRICAL  EVALUATION  OF 


THREE  KNOWLEDGE  ACQUISITION  TECHNIQUES  FOR  DEVELOPING 
A  PROJECT  MANAGEMENT  RELATED  EXPERT  SYSTEM 

I.  Introduction 


General  Issue 

Advances  in  computer  software  technology,  coupled  with 
the  growth  of  microcomputer  hardware,  have  made  computer- 
aided  decision  support  and  expert  systems  available  to 
virtually  all  Air  Force  managers.  However,  one  of  the 
fundamental  difficulties  encountered  when  developing  an 
expert  system  is  not  the  software  or  hardware,  but  acquiring 
the  expert  knowledge  that  must  be  incorporated  into  the 
system.  This  knowledge  must  be  extracted  from  human  experts 
in  sufficient  quantity  and  quality  so  that  it  can  be 
translated  and  programmed  into  a  computer.  While  many 
knowledge  acquisition  techniques  exist,  little  if  any 
empirical  evidence  exists  to  indicate  which  knowledge 
acquisition  techniques  may  be  preferred  for  specific  subject 
areas  (often  referred  to  as  "knowledge  domains"). 

Spec i f ic  Purpose 

The  purpose  of  this  study  is  to  identify  and  evaluate 
the  relative  effectiveness  of  three  knowledge  acquisition 
techniques  that  may  be  used  when  developing  expert  systems 
for  project  management  related  tasks. 
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Research  Object i ves 


With  the  many  knowledge  acquisition  techniques 
available,  one  technique  or  combination  of  techniques  should 
prove  effective  in  aiding  the  development  of  a  work 
breakdown  structure  microcomputer  expert  system. 

The  following  are  the  research  objectives  of  this 
thesis: 

1.  Understand  the  nature  of  the  work  breakdown 
structure  development  process. 

2.  Identify  current  knowledge  acquisition  techniques 
being  used  in  the  field  of  expert  system  technology. 

3.  Select  three  knowledge  acquisition  techniques  to 
use  in  extracting  knowledge  to  develop  an  expert  system  to 
aid  the  creation  of  a  project  work  breakdown  structure. 

4.  Field  test  these  three  knowledge  acquisition 
techniques. 

5.  Analyze  the  results  of  the  field  test  of  the  three 
knowledge  acquisition  techniques. 

6.  Develop  a  simple  expert  system  based  on  the 
techniques. 

7.  Val idate  the  expert  system  to  see  how  well  it 
reflects  the  knowledge  of  the  experts  who  participated  in 
this  evaluation. 
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Assumpt ions  and  Standards 

Knowledge  acquisition  is  recognized  as  the  "bottleneck" 
in  the  burgeoning  field  of  expert  system  technology  (3:144). 
Even  though  many  acquisition  techniques  have  been 
identified,  the  difficulty  lies  in  selecting  a  technique 
that  meets  four  essential  criteria: 

1.  It  is  a  technique  appropriate  for  the  knowledge 
being  collected; 

2.  It  is  relatively  easy  for  the  system  programmer  (or 
knowledge  engineer)  to  use; 

3.  It  produces  a  body  of  knowledge  in  a  form  that  does 
not  require  complex  interpretat ion  (knowledge  that  is 
unambiguous);  and 

4.  It  produces  the  knowledge  in  a  form  that  can  be 
encoded  in  the  appropriate  decision  rules  required  for  the 
computer . 

Sc  ope  and  L l mi t  at i ons  of  This  Research 

The  scope  of  this  research  is  limited  to  the 
fundamental  considerations  affecting  knowledge  acquisition. 
Knowledge  acquisition,  as  used  in  this  study,  is  the 
collection  and  translation  of  an  expert's  knowledge  into  a 
format  suitable  for  a  particular  expert  system.  The 
difficulty  associated  with  this  task  lies  in  the  difference 
between  the  representat ional  format  required  for  the  expert 
system  and  the  basic  form  of  the  knowledge  itself  (10:168). 
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This  research  examines  the  extraction  of  knowledge  for 
one  particular  problem  area  or  domain:  project  management. 
Due  to  time  constraints,  not  all  knowledge  acquisition 
techniques  or  problem  tasks  could  be  investigated.  As  such, 
one  representat i ve  task,  that  of  project  work  breaxdown 
structure  development,  was  the  goal  of  the  expert  system, 
and  three  k.  wledge  acquisition  techniques  were  evaluated. 
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II. 


Methodol ogy 


Overv i ew 

As  stated  in  Chapter  I,  the  acquisition  of  expert 
knowledge  is  one  of  the  primary  difficulties  facing  the 
expert  system  programmer  or  knowledge  engineer.  This 
difficulty  is  compounded  by  the  fact  that  few  guidelines 
exist  to  help  the  expert  system  programmer  with  the 
acquisition  of  expert  knowledge  (5:155).  The  goal  of  this 
research  is  to  help  an  expert  system  programmer,  whether  a 
novice  or  an  accomplished  knowledge  engineer,  identify  and 
implement  an  appropriate  knowledge  acquisition  technique  for 
the  task  at  hand. 

As  a  basis  for  this  research,  a  detailed  literature 
review  was  conducted  examining  expert  system  technology  and 
the  expert  system  application  problem  domain  selected  for 
this  research.  Chapter  III  provides  an  introduction  to  the 
knowledge  acquisition  techniques  evaluated  by  this  research 
and  a  summary  of  relevant  information  on  the  problem  domain: 
project  management.  A  separate  report,  titled  "An 
Introduction  to  Expert  Systems  and  Knowledge  Acquisition 
Techniques"  was  written  as  a  result  of  this  1 iterature 
review.  "An  Introduction  to  Expert  Systems  and  Knowledge 
Acquisition  Techniques"  provides  a  general  description  of 
expert  systems,  expert  knowledge,  and  knowledge  acquisition. 
This  report  was  published  separately  (9). 
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The  1 iterature  reviews  examined  text  books, 
professional  journals,  other  theses,  Air  Force  regulations, 
and  military  standards. 

The  following  paragraphs  explain  the  methods  used  to 
address  the  seven  research  objectives  outlined  in  Chapter  I. 

Object  ive  1_:  Understand  the  nature  of  the  work 
breakdown  structure  development  process.  A  literature 
review  provided  the  basis  for  an  understanding  of  the  use 
and  development  of  a  work  breakdown  structure.  Chapter  III 
provides  a  summary  of  this  information.  The  background 
1 iterature  consisted  of  textbooks  on  project  management,  Air 
Force  regulations,  Air  Force  pamphlets,  and  Military 
Standard-881 ,  which  deals  specifically  with  the  work 
breakdown  structure. 

The  1 iterature  review  wcs  supplemented  by  discussions 
with  those  knowledgeable  in  the  work  breakdown  structure 
process.  These  individuals  are  assigned  to  the  Air  Force 
Institute  of  Technology  and  Aeronautical  Systems  Division 
(ASD)  at  Wr ight-Patterson  AFB,  OH.  These  individuals 
provided  insight  into  how  the  Air  Force,  and  more 
specifically,  how  ASD  implements  the  work  breakdown 
structure  for  its  many  projects  and  programs. 

Object  ive  2_:  I  dent  i  f  y  current  knowl  edge  acquisit  ion 

techniques  being  used  in  the  field  of  expert  system 
technol oov.  A  detailed  1 iterature  review  was  conducted  to 
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determine  what  knowledge  acquisition  techniques  are 
currently  being  used  in  the  field  of  expert  system 
technology.  This  literature  review  resulted  in  the 
publication  of  the  companion  report  titled  “An  Introduction 
to  Expert  Systems  and  Knowledge  Acquisition  Techniques"  (9). 
This  companion  report  provides  an  overview  of  expert  system 
technology,  expert  knowledge,  a  brief  definition  of  each 
recognized  knowledge  acquisition  technique,  the  types  of 
knowledge  that  these  techniques  were  reported  to  be 
appropriate  for,  and  recognized  limitations  of  each 
techn ique. 

Object  i  ve  3_:  Select  three  knowl  edge  acquisition 
techniques  to  use  in  extract ino  knowl edge  to  devel op  an 
expert  system  to  aid  in  the  creat  ion  of  a_  project  work 
breakdown  structure.  From  the  selection  of  recognized 
knowledge  acquisition  techniques  found  in  the  literature, 
three  techniques  were  selected  for  evaluation.  The  three 
techniques  were  interview,  concept  mapping,  and  interruption 
analysis  (task  observation).  Each  of  these  techniques  is 
described  in  Chapter  III.  In  general,  these  techniques  were 
selected  based  on  the  opinions  of  the  1 iterature  authors  as 
to  the  merits  of  each  technique  for  the  selected 
application,  and  the  type  of  knowledge  that  each  technique 
was  suitable  to  acquire. 

Information  presented  in  the  literature  indicated  that 
each  of  these  knowledge  acquisition  techniques  was 
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potentially  suitable  for  acquiring  knowledge  about  a  task 
where  a  verbal  description  of  the  expert's  problem  solving 
process  was  possible.  Based  on  discussions  with  individuals 
knowledgeable  about  work  breakdown  structures,  the  task  of 
developing  a  work  breakdown  structure  was  assumed  to  be 
expl a inabl e. 

Object  i  ve  4_:  Field  t est  t hese  three  knowl  edge 
ac gu i s i t i on  techniques.  A  quasiexper iment  was  conducted  to 
assess  the  relative  effectiveness  of  the  three  selected 
techniques.  Each  knowledge  acquisition  technique  was 
evaluated  using  a  different  expert. 

The  process  of  identifying  experts  involved  discussions 
with  individuals  from  Aeronautical  Systems  Division's  Cost 
Management  Systems  Division  of  the  Directorate  of  Cost 
(ASD/ACCM).  LtCol  Thomas  Bowman  recommended  using  experts 
from  a  program  office  cost  analysis  directorate.  The 
Directorate  of  Program  Control  (ASD/RWP)  from  the  Electronic 
Combat  and  Reconnaissance  Systems  Program  Office  was 
contacted.  The  office  expressed  an  interest  in  this 
research  project  and,  more  importantly,  a  willingness  to 
part ic ipate. 

Individual  experts  were  identified  through  personal 
interviews  and  recommendations  from  peers  within  the  cost 
analysis  directorate.  Besides  their  knowledge,  the  two 
dominant  character ist ics  desired  of  the  experts  were  a 
willingness  to  participate  in  this  research  and  assured 
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availability.  Experts  volunteered  to  participate  in  this 
exper iment . 

Three  approximately  one-hour  sessions  were  used  to 
extract  each  expert's  knowledge.  A  fourth  meeting  was  used 
to  review  the  information  collected  during  the  course  of  the 
previous  sessions  to  identify  any  errors  or  omissions.  Each 
session  was  audio-taped  in  addition  to  any  written  notes 
taken.  Each  session  was  conducted  at  a  location  that  was 
convenient  and  comfortable  to  each  expert. 

Before  the  actual  knowledge  acquisition  began,  the 
investigator  practiced  each  knowledge  acquisition  technique. 
The  practices  were  conducted  using  experts  in  the  same 
general  field  as  those  participating  in  the  experiment. 
One-hour  practice  sessions  involving  the  interview  and  the 
interruption  analysis  techniques  were  conducted.  The 
interruption  analysis  practice  was  conducted  during  an 
actual  program  initiation  meeting  involving  the  expert  who 
participated  in  the  interview  evaluation.  However,  the 
interruption  analysis  practice  took  place  after  the  three 
interview  sessions  were  completed.  A  two-hour  practice 
concept  mapping  session  was  conducted. 

Object ive  5:  Analyze  the  resul t s  of  the  field  test  of 
the  three  knowl edge  acqui si t ion  techniques.  Six 
quantitative  measures  were  identified  as  a  means  to  assess 
the  relative  effectiveness  of  the  three  knowledge 
acquisition  techniques.  These  measures  were  as  follows: 
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1.  The  total  time  required  to  implement  each 
technique.  This  included  actual  interaction  time  with  the 
expert,  preparation  time,  and  the  time  required  to 
transcribe  and  integrate  information  from  the  audio-tapes 
and  notes  into  a  single  written  transcript. 

2.  The  number  of  rules  and  facts  produced. 

3.  The  time  required  to  formulate  rules  from  the 
written  transcripts. 

4.  The  number  of  inferences  required  to  complete 
rules. 

5.  Each  technique’s  contribution,  measured  in  the 
number  of  rules,  to  the  final  expert  system. 

6.  The  number  of  intermediate  steps  required  to 
translate  the  expert’s  knowledge  into  an  expert  system  rule. 

Object i ve  6^  Devel op  a_  simpl e  expert  system  based  on 
in  format  ion  collected  from  each  of  the  three  techniques.  As 
stated  in  Item  6  in  the  previous  objective,  a  simple  expert 
system  was  created  using  the  knowledge  from  each  of  the 
three  experts. 

Ob jec t  i  ve  7_:  Val  idate  the  expert  system  to  see  how 
wel 1  i t  re f 1 ected  the  knowl edge  of  the  experts  who 
part ic ipated  i n  this  pro jec t .  To  validate  the  expert 
system,  the  system  was  demonstrated  to  the  part ic ipat ing 
experts.  Each  expert  was  asked  to  solve  a  simple  test  case 
using  the  expert  system.  The  expert  system’s  decision  rules 
and  output  were  evaluated  to  determine  if  the  problem 
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solving  methodology  and  solutions  to  the  case  were  the  same 
as  those  expected  by  the  experts.  Discrepanc ies  and  errors 
were  traced  to  determine  if  they  resulted  from  errors  in 
programming,  unfounded  assumptions  on  the  part  of  the  system 
developer  that  were  the  consequence  of  incomplete 
information,  or  actual  errors  in  the  experts'  thought 
processes  when  they  communicated  the  information  to  the 
system  developer. 

Summary 

The  purpose  of  this  research  effort  was  to  evaluate 
knowledge  acquisition  techniques  within  the  context  of 
expert  system  development.  Knowledge  extracted  by  three 
recognized  knowledge  acquisition  techniques  was  evaluated  to 
determine  if  the  form  and  content  of  the  data  could  be 
readily  used  to  develop  the  expert  system  knowledge  base 
rules  and  facts.  An  expert  system,  created  as  a  part  of 
this  research,  was  evaluated  by  the  participating  experts  to 
ensure  that  the  system  produced  acceptable  output  and  to 
determine  the  source  of  any  errors  that  may  have  occurred. 


III.  Literature  Review 

Topic  Statement 

This  chapter  reviews  the  1 iterature  on  knowledge 
acquisition  processes  that  were  potentially  useful  in 
developing  an  expert  system  knowledge  base  to  aid  the 
development  of  a  project  work  breakdown  structure.  An 
expert  system  is  a  computer  system  that  emulates  or  copies 
the  human  cognitive  decision  and  problem  solving  process 
(12:395).  As  stated  previously,  h i st or ical 1 y ,  the 
acquisition  of  expert  knowledge  has  caused  difficulty  during 
the  development  of  expert  systems.  While  this  obstacle  is 
acknowledged  as  one  of  the  primary  difficulties  encountered 
in  expert  system  development,  there  is  no  universally 
accepted  reason  why  (3:144). 

Method  o f  Treat ment  and  Or gan i 2  at i on 

This  1 iterature  review  contains  literature  that 
supports  an  empirical  evaluation  of  the  knowledge 
acquisition  techniques  available  to  the  expert  system 
developer  or  knowledge  engineer.  The  primary  goal  of  this 
1 iterature  review  is  to  acquaint  the  reader  with  the 
knowledge  acquisition  techniques  used  in  this  research  and  a 
specific  task  that  this  technology  may  be  applied  to.  It 
provides  an  overview  of  knowledge  acquisition,  and  specific 
definitions  and  methodologies  for  the  three  knowledge 
acquisition  techniques  evaluated  by  this  research.  It  also 
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provides  an  overview  of  a  specific  problem  domain,  and 
identifies  and  describes  a  particular  task  within  that 
domain  that  is  suitable  for  expert  system  application. 

Resourc  es 

The  literature  used  for  this  research  effort  consisted 
of  professional  journals,  technical  publications,  Air  Force 
regulations  and  military  standards,  theses,  and  textbooks 
about  expert  systems,  artificial  intelligence,  and  project 
management.  Although  the  authors  writing  about  artificial 
intelligence  ana  expert  systems  discussed  many  facets 
associated  with  these  fields,  a  common  thread  throughout 
much  of  the  literature  was  the  difficulty  surrounding  the 
acquisition  of  expert  knowledge  (3:144;  15:152). 

Knowl edge  Acquisition 

"The  elicitation  of  knowledge  from  experts  is  a  time- 
consuming  process  and  is  usually  conducted  in  the  absence  of 
a  systematic  conceptual  design.  Few  guidelines  are 
available  to  help  the  knowledge  engineer  map  out  his/her 
course  and  pursue  it  efficiently"  (5:155).  There  are  many 
difficulties  associated  with  developing  an  adequate 
knowledge  base  for  an  expert  system.  These  difficulties  can 
be  attributed  to  the  diversity  of  knowledge  forms,  and  the 
difficulty  of  managing  the  knowledge  data  base  (1:103).  The 
primary  difficulty  facing  the  knowledge  engineer  is 
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accessing  the  “...abstract  generalizations  of  the  expert’s 
knowledge  domain"  (1:123). 

Within  the  context  of  expert  system  technology, 
knowledge  acquisition  is  the  collection  and  translation  of 
expert  problem  solving  ability  from  a  knowledge  source  to  a 
computer  program  (2:129).  01  son  identifies  two  general 

classes  of  acquisition  techniques — direct  methods  and 
indirect  methods.  Direct  methods  focus  on  the  knowledge 
that  can  be  directly  articulated,  explicit  knowledge. 

Direct  methods  include,  among  others,  interviews, 
questionnaires,  protocol  analysis,  interruption  analysis, 
task  observation,  drawing  closed  curves  and  inferential  flow 
analysis  (15:153-166). 

Belkin  states  that  these  "direct  methods"  can  be 
grouped  into  the  main  acquisition  techniques  of  the 
interview,  verbal  protocol  analysis,  and  observational 
studies.  Interviews  may  be  either  structured  or  informal. 
Verbal  protocol  analysis  is  analyzing  the  problem  solving 
process  as  experts  verbalize  their  thought  processes  while 
solving  a  task.  Observat ional  studies  are  data  collection 
efforts  for  actual  problems  in  the  expert’s  natural  working 
environment.  According  to  Belkin,  current  practice  is  to 
use  a  combination  of  techniques,  and  certain  techniques  may 
be  more  suitable  for  specific  types  of  knowledge  (2:129- 
130)  . 
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According  to  Olson,  indirect  methods  collect 
information  that  requires  inferences  to  be  made  about  the 
exact  nature  of  the  expert’s  knowledge.  Indirect  methods 
attempt  to  extract  implicit  expert  knowledge.  Indirect 
methods  include  multi-dimensional  scaling,  Johnson’s 
hierarchical  scaling,  general  weighted  networks,  ordered 
trees  from  recall,  and  repertory  grid  analysis  (15:153-166). 

These  lists  of  direct  and  indirect  knowledge 
acquisition  techniques  do  not  represent  all  of  the  various 
techniques  currently  being  used.  They  are  merely  an  attempt 
to  convey  to  the  reader  the  great  variety  of  forms  available 
to  the  knowledge  engineer.  Other  methods  not  found  to 
easily  fit  within  the  previous  two  categories  include 
concept  mapping  or  visual  modelling,  discourse  analysis,  and 
storyboard ing .  The  most  common  knowledge  acquisition 
method,  however,  is  the  interview  (15:153).  Olson 
recommends  beginning  any  knowledge  acquisition  effort  with 
an  unstructured  interview  (15:154). 

Olson  goes  on  to  caution  that  no  matter  which  technique 
or  techniques  are  used,  each  acquisition  technique  (direct 
or  indirect)  may  lead  to  incorrect  rules  and  relations. 

"The  knowledge  engineer  must  make  Judgments  of  the 
suitability  of  a  method  for  knowledge  elicitation  for  the 
kinds  of  knowledge  the  expert  is  assumed  to  possess" 

(15: 167) . 
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While  there  exist  many  different  knowledge  acquisition 


techniques  described  throughout  the  literature,  this 
research  focused  on  the  interview,  concept  mapping,  and 
interruption  analysis.  This  was  done  because  these  three 
knowledge  acquisition  techniques  appeared  to  meet  the 
"Assumptions  and  Standards"  as  set  forth  in  Chapter  I  of 
this  report.  Those  three  techniques  are  discussed  below, 
and  all  techniques  assessed  during  the  research  are 
discussed  in  the  companion  volume  to  this  report  (9). 

Interviewing .  According  to  Olson,  the  interview  is  the 
most  common  knowledge  acquisition  technique  (15:153).  In 
essence,  "the  exchange  of  information  is  the  central  purpose 
of  the  interview"  <Q:54).  The  interview  involves 
conversational  interaction  between  individuals  to  convey 
information  about  a  particular  subject.  The  interview  is 
considered  a  direct  method  of  knowledge  acquisition.  In 
other  words,  the  interview  is  best  suited  to  knowledge  that 
can  be  articulated  (15:153). 

In  general  terms,  interviews  can  be  classified  as 
either  standardized  (structured)  or  non-st andard i zed 
(unstructured) .  A  standardized  interview  seeks  to  gather 
the  same  information  from  a  number  of  different  respondents. 
To  insure  this  uniformity,  the  same  questions  are  asked  of 
each  respondent.  The  structured  interview  can  be  further 
divided  into  scheduled  interviews,  where  the  questions  are 
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asked  in  the  same  order,  or  non-scheduled  interviews,  where 
the  order  and  wording  of  the  questions  may  be  varied  (8:60). 

Non-standardized  interviews  do  not  pose  all  the  same 
questions  to  all  of  the  respondents.  As  such,  answers 
cannot  be  statistically  manipulated  to  analyze  group 
responses  or  to  compare  individual  responses  to  the 
aggregate.  Unstructured  interviews  can  be  classified  as 
preparatory  unstructured  interviews  which  are  done  to 
prepare  for  a  structured  interview,  or  independent 
unstructured  interviews  which  are  not  preparatory  for 
additional  interviews  and  have  their  own  purpose  (8:61). 

Concept  Mapping.  Within  the  scope  of  expert  system 
development,  concept  mapping  can  be  used  as  a  complement  to 
an  interview.  Concept  maps  are  a  visual  representat ion  of 
hierarchical  relationships  among  concepts.  "Concept  maps 
are  intended  to  represent  meaningful  relationships  between 
concepts  in  the  form  of  propos i t i ons "  (14:15).  Concept 
mapping  is  potentially  useful  in  extracting  implicit  expert 
knowledge  since  it  focuses  on  concepts  and  relationships 
among  those  concepts. 

As  stated  above,  concept  mapping  is  used  in  conjunction 
with  interviewing  techniques  to  produce  a  graphical 
facsimile  of  an  expert’s  knowledge.  As  such,  considerations 
given  to  proper  interviewing  should  be  applied  when 
employing  the  concept  mapping  technique.  Unlike  an 
interview,  however,  the  end-product  of  a  concept  mapping 
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session  will  not  be  a  detailed  transcript  but  a  hierarchical 
concept  map  of  the  expert's  knowledge  (14:119-133).  Each 
mapping  session  is  guided  by  and  builds  upon  the  previous 
session's  concept  map.  Figure  1  is  a  simple  example  of  a 
concept  map. 


have 


can  ba 


Figure  1.  Example  of  a  Simple  Concept  flap. 

(Reprinted  from  14:18) 

Interrupt  ion  Anal / sis.  As  described  in  Olson's 
article,  interruption  analysis  is  a  method  for  accessing 
explicit  forms  of  expert  knowledge.  It  is  similar  .o  simple 
task  observation.  The  knowledge  engineer  observes  an  expert 
solving  a  specific  task.  Unlike  task  observation,  however, 
interruption  analysis  allows  the  knowledge  engineer  to 
interrupt  the  problem  solving  process  when  the  expert  does 
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something  the  knowledge  engineer  cannot  understand.  At  this 
point,  the  reasoning  process  and  actions  of  the  expert  are 
examined  and  the  expert  tries  to  explain  in  detail  the 
reasons  for  his  actions  (15:156). 

Olson  feels  the  main  advantage  of  interruption  analysis 
is  that  the  knowledge  engineer  is  able  to  capture  an 
expert's  knowledge  at  the  critical  moment  of  execution  and 
theoretically,  at  the  moment  of  greatest  focus.  This 
advantage  is  somewhat  compromised  in  that  the  thought 
processes  are  interrupted,  and  they  may  prove  impossible  to 
restart.  Olson  feels  this  technique  provides  the  best 
results  when  it  is  employed  after  a  prototype  expert  system 
has  been  developed,  and  the  knowledge  engineer  wishes  to 
compare  the  expert  system's  performance  to  that  of  an  expert 
(15: 156). 

Pr ob 1 em  Doma i n  Overview 

A  suitable  application  for  expert  system  technology  is 
one  where  there  is  an  "existing  body  of  expertise,  and  this 
expertise  is  routinely  used  for  decision  making..." 

(16:106).  Using  this  1 iberal  interpretat ion,  almost  any 
task  would  appear  to  be  suitable.  However,  if  the  opinions 
of  others  familiar  with  this  technology,  such  as  Waterman, 
Kiem  and  Jacobs,  and  Prerau,  are  taken  into  consideration, 
some  explicit  criteria  exist  by  which  to  determine  if  a 
problem  domain  is  suitable  for  expert  system  application.  A 
more  detailed  description  of  the  criteria  that  can  be  used 


19 


to  identify  suitable  applications  for  expert  system 
technology  can  be  found  in  "An  Introduction  to  Expert 
Systems  and  Knowledge  Acquisition  Techniques"  (9).  One 
problem  domain  that  appears  to  meet  most  of  the  existing 
criteria  is  project  management. 

Project  management  is  a  multifaceted  discipline.  As 
such,  one  working  under  these  c ircumstances  "is  faced  with 
many  non-routine  and  unstructured  decisions"  (4:1).  If  a 
project  is  defined  as  a  specific  task  with  a  specific 
objective,  start  and  end  dates,  limited  funding  and  resource 
consumption,  project  management  involves  the  planning  and 
monitoring  of  that  project  (11:2).  Kerzner  defines  project 
planning  as  defining  work  requ irements,  defining  the 
quantity  of  work,  and  determining  the  necessary  resources 
(11:2).  Project  monitoring  is  overseeing  progress, 
comparing  actual  progress  to  predicted  progress,  assessing 
the  impact  of  this  comparison,  and  adjusting  to  subsequent 
changes  (11:2-3).  According  to  Kerzner,  "the  major 
responsibility  of  the  project  manager  is  planning"  (11:18). 

"Planning  in  a  project  environment  may  be  described  as 
establishing  a  predetermined  course  of  action  within  a 
forecasted  environment"  (11:575).  Establishing  this 
predetermined  course  of  action  involves  many  individual,  yet 
interrelated  tasks  such  as  determining  the  life  cycle  phases 
of  the  project,  general  project  planning,  generating  a 
statement  of  work  (SOW),  scheduling,  and  developing  the  work 
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breakdown  structure.  The  development  of  the  work  breakdown 
structure  is  but  one  potential  application  for  expert  system 
technology,  and  is  the  focus  of  this  research. 

The  Nor k  Breakdown  Structure.  The  work  breakdown 
structure  "is  a  product-or iented  family  tree  composed  of 
hardware,  services,  and  data  which  result  from  project 
engineering  efforts  during  the  development  and  production  of 
a  defense  materiel  item,  and  which  completely  defines  the 
projec t /program"  (7:11-34).  A  sample  work  breakdown 
structure  tree  diagram  is  presented  in  Figure  2.  A  project 
manager  should  define  all  project  work  in  elements  that  are 
manageable,  independent,  integratabl e,  and  measurable 
(11:599).  The  work  breakdown  structure  provides  a  framework 
by  which  a  project  manager  can  accomplish  this  breakdown. 

The  work  breakdown  structure  may  follow  a  format  similar  to 
that  provided  in  Figure  3. 

Air  Force  Regulation  800-17  states  the  policy  for 
implementing  a  work  breakdown  structure  for  Air  Force 
acquisition  programs.  This  regulation  applies  to  all 
organizat ions  "which  wish  to  issue  requests  for  proposals 
for  defense  materiel  items..."  (6:1).  According  to  this 
regulation,  the  work  breakdown  structure  provides  a  somewhat 
standardized  format  against  which  all  acquisition  programs 
can  be  broken  down  into  the  products  and  services  that  make 
up  a  particular  defense  materiel  item.  The  objective  of 
this  regulation  is  to  ensure  that  the  top  three  levels 
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Figure  2.  Example  of  a  Work  Breakdown  Structure 
Tree  Diagram  (Adapted  from  11:607) 
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Figure  3. 

Generic  Work  Breakdown  Structure 
(Reprinted  from  11:600) 
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of  a  work  breakdown  structure  conform  to  a  framework  that 
allows  accurate  comparisons  of  similar  acquisition  programs. 

Military  Standard  881  (MIL-STD-881 )  is  the  document 
that  directs  the  development  of  a  work  breakdown  structure 
for  all  defense  materiel  items.  MIL-STD-881  provides  an 
introductory  narrative  explaining  the  purpose  and  various 
types  of  work  breakdown  structures.  It  also  has  seven 
appendices  that  provide  generic  work  breakdown  structures 
and  element  definitions  for  the  top  three  levels  of  the 
major  categories  of  defense  materiel  items.  These 
categories  are: 

1.  Aircraft  Systems 

2.  Electronic  Systems 

3.  Missile  Systems 

4.  Ordnance  Systems 

5.  Ship  Systems 

6.  Space  Systems 

7.  Surface  Vehicle  Systems 

The  work  breakdown  structure  is  a  fundamental  tool  for 
effective  project  management.  Part  of  this  research  to 
assess  the  relative  effectiveness  of  the  knowledge 
acquisition  techniques  was  the  development  of  an  expert 
system  to  assist  the  development  of  a  project  work  breakdown 
structure.  However,  since  the  expert  system  was  not  the 
goal  of  this  research,  the  expert  system  produced  during  the 
course  of  this  research  is  a  limited  prototype. 
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IV.  Results  and  Findings 


Overview 

This  chapter  presents  the  results  of  the  evaluation  of 
the  three  knowledge  acquisition  techniques:  the  interview, 
concept  mapping,  and  interruption  analysis.  The  results  of 
this  quasiexperiment  are  broken  down  into  three  parts: 
technique  implementation,  rule  and  fact  production,  and 
expert  system  validation. 

General  Comments 

This  experiment  attempted  to  evaluate  one  complicated 
facet  of  the  expert  system  development  process.  Several  of 
the  fundamental  steps  necessary  to  execute  the  evaluation  of 
the  knowledge  acquisition  techniques  are  provided  in  the 
methodology  and  not  in  this  chapter.  This  was  not  done 
because  these  particular  steps  are  less  important  or  easier 
than  the  evaluation  of  the  knowledge  acquisition  techniques. 
On  the  contrary,  each  step  was  vitally  important  to  the 
success  of  this  evaluation.  This  chapter  is  reserved 
specifically  for  the  results  of  evaluating  the  knowledge 
acquisition  techniques.  The  reader  is  encouraged  to  review 
the  process  of  task  selection,  the  selection  of  experts,  and 
the  selection  of  the  techniques  to  be  evaluated  presented  in 
Chapter  III.  Much  of  this  work  is  also  supported  by  the 
literature  reported  in  "An  Introduction  to  Expert  Systems 
and  Knowledge  Acquisition  Techniques”  (9). 
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Techn igue  Impl ementat ion 


The  results  presented  in  this  section  refer  to  that 
portion  of  this  experiment  that  dealt  with  executing  each  of 
the  knowledge  acquisition  techniques.  It  includes 
preparation  time,  the  time  spent  interacting  with  the 
expert,  and  the  time  it  took  to  translate  the  initial 
knowledge  into  a  form  that  could  be  used  for  rule  and  fact 
generat ion . 

Knowl edge  Acouisit ion  Tec hn i que  Impl ementat ion .  As  the 
reader  will  recall,  each  knowledge  acquisition  session  was 
scheduled  for  appr ox i mat el y  one  hour.  In  addition  to  the 
actual  time  spent  interacting  with  the  experts,  the  time 
spent  transcribing  and  integrating  information  from  the 
audiotapes  and  other  sources,  as  well  as  the  total  time 
involved  in  preparing  for  each  technique  was  recorded. 

Table  1  presents  the  results  of  this  phase  of  the 
evaluation.  These  results  show  that  the  interview  technique 
took  the  least  amount  of  time  (419.1  mins),  followed  by 
interruption  analysis  <505.4  mins),  and  concept  mapping 
(630.0  mins).  These  total  times  disguise  the  true  time 
consuming  factor,  that  being  the  transcr ipt ion  times.  Both 
the  interview  and  interruption  analysis  had  comparable 
transcr ipt ion  times,  yet  the  concept  mapping  took  almost 
twice  as  long  to  transcribe  as  the  other  two  techniques. 

The  t  r  ansc  r  i  pt  i  on  times  account  for  497.  of  the  total 
interview  time,  427.  of  the  total  interruption  analysis  time 
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Table  1. 

Knowledge  Acquisition  Technique 
Implementation  Times’ 


Interaction 

T  imes 

Transcription 

T  imes 

Preparat ion 
Times 

Total 

Time 

Interview 

101.4 

205.0 

32.0 

419.1 

Concept 

Mapping 

166.0 

404.0 

60.0 

630.0 

Interruption 

Analysis 

188.8 

211.3 

105.3 

505.4 

’All  times  in  minutes. 


and  647.  of  the  concept  mapping  time.  It  should  be  noted, 
however,  that  the  audiotapes  of  the  interviews  were  not 
transcribed  "word-for-word. "  Detailed  notes  of  the 
interview  tapes  were  prepared  in  place  of  the  "word-for- 
word"  transcript.  Appendix  A  provides  a  sample  of  a  "word 
for-word"  transcr ipt ion  of  a  portion  of  one  interview.  If 
the  interview  tapes  had  been  transcribed  "word-for-word,  " 
considerably  more  time  would  have  accumulated  during  the 
transcr ipt ion  portion  of  this  test. 

The  preparation  times  show  a  variation  of  75  minutes 
between  the  two  extremes.  The  interview  preparation  time 
consisted  of  identifying  questions  to  start  each  interview 
session.  The  majority  of  this  time  occurred  before  the 
first  interview,  with  subsequent  questions  arising  during 
the  transcr ipt ion  of  the  interview  tapes.  The  preparation 
time  for  the  concept  mapping  was  spent  teaching  the  expert 
about  the  concept  mapping  technique,  and  having  the  expert 
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generate  some  unrelated  practice  concept  maps.  Questions 
used  during  the  actual  concept  mapping  sessions  were 
generated  as  individual  maps  were  drawn.  For  the 
interruption  analysis,  the  preparation  time  was  spent 
identifying  and  developing  two  case  studies.  One  case  study 
was  available  from  the  AFIT  Acquisition  and  Planning  SYS  200 
course  text  (AFIT/LS).  The  second  case  study  was  developed 
by  this  researcher  and  accounts  for  the  bulk  of  this 
preparation  time. 

Rul e  and  Fact  Pr oduc  t i on 

This  portion  of  the  evaluation  examined  five  different 
criteria  as  quantitative  measures  of  effectiveness  for  each 
knowledge  acquisition  technique.  These  quantitative 
measures  were: 

1.  The  time  required  to  formulate  rules; 

2.  The  total  number  of  rules  generated; 

3.  The  number  of  inferences  necessary  to  complete 
rules; 

4.  An  examination  of  the  different  knowledge  sets  to 
determine  the  contribution  each  knowledge  acquisition 
technique  made  to  the  final  expert  system  developed  during 
the  course  of  this  research;  and 

5.  The  number  of  translations  necessary  to  convert  the 
expert  knowledge  into  reasonably  correct  expert  system 

rul es . 
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In  addition  to  these  quantitative  measures,  each 
knowledge  acquisition  technique  was  rated  on  the  ease  with 
which  the  data  could  be  translated  into  expert  systems 
rules. 

Rul e  Formul at  ion  T imes.  Rule  formulation  time  was 

measured  because  time  is  a  valuable  resource  for  the 

knowledge  engineer,  and  all  things  being  equal,  the 

technique  that  allowed  the  quickest  formulation  of  rules 

would  be  considered  the  more  desirable  knowledge  acquisition 

technique.  For  this  measure,  the  work  breakdown  structure 

development  process  was  divided  into  three  relatively 

discrete  .ions,  the  first  dealing  with  the  program 

hardware,  the  second  dealing  with  elements  referred  to  as 

"o*- her  contract  costs,"  and  the  third  dealing  with  "other 

government  costs. "  A  random  number  generator  was  used  to 

match  each  knowledge  acquisition  technique  to  a  particular 

portion  of  the  work  breakdown  structure  process.  Then, 

three  production  rules  were  generated  from  each  knowledge 

data  set  for  the  portion  of  the  work  breakdown  structure 

process  to  which  it  was  matched.  A  production  rule  is  a 

rule  that  follows  the  "If-Then"  format ,  but  in  a  narrative 

form  that  does  not  necessarily  match  the  syntax  required  for 

the  expert  system  software. 

IF:  This  is  a  development  work  breakdown 

structure,  AND  other  contract  costs  are  to  be 
considered,  AND  Air  Force  experience  with  this 
prime  mission  equipment  is  limited; 
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THEN:  Type  1  Training  should  be  included  as  a 

level  2  work  breakdown  structure  element  of  the 
Full-Scale  Development  work  breakdown  structure. 


The  times  required  to  formulate  each  rule  are  recorded  in 
Table  2. 


Table  2. 


Rule  Formulation  Times* 

Rules 

First 

Second 

Third 

Total 

Average 

Interview 

8.43 

2.22 

2.67 

13.32 

4.44 

Concept 

3.42 

2.22 

2.67 

8.31 

2.77 

Mapping 

Interrupt  ion 

4.53 

7.67 

5.70 

17.90 

5.97 

Analysis 

*  All  time  in  minutes 

Clearly,  concept  mapping  required  the  least  time  to 
formulate  production  rules  for  an  expert  system.  Also, 
concept  mapping  demonstrated  the  least  variation  in  time  for 
its  three  rules  among  the  three  techniques.  One  could  argue 
that  the  interview  total  was  skewed  because  of  the  time 
required  for  the  first  rule.  This  time  resulted  from  having 
to  search  through  all  of  the  interview  material  to  find 
references  for  the  pertinent  portion  of  the  work  breakdown 
structure  process.  Mhile  it  appears  that  the  subsequent 
formulation  times  approach  those  of  the  concept  map, 
searching  for  the  unrelated  bits  of  information  collected  by 
the  interview  technique  could  be  expected  to  increase  times 
as  more  complex  rules  were  formulated.  Additionally,  a 
longer  initial  search  time  to  identify  pertinent  data  could 
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be  expected  each  time  a  different  portion  of  the  work 
breakdown  structure  process  was  attempted. 

Rule  F ormul at  ion  and  Inferences  Required.  The  number 
of  rules  formulated  and  the  inferences  required  to  formulate 
those  rules  were  measured  because  experts  are  typically  very 
busy  individuals  and  as  such,  it  is  better  to  acquire  more 
pertinent  knowledge  per  unit  of  interaction  time. 
Additionally,  the  fewer  the  inferences  the  knowledge 
engineer  is  required  to  make,  the  more  accurately  the  expert 
system  rule  base  will  reflect  the  expert's  knowledge. 

Again,  the  work  breakdown  structure  process  was  separated 
into  three  discrete  portions  as  was  done  for  the  previous 
test.  A  random  number  generator  was  used  to  select  one  of 
these  three  discrete  portions.  All  of  the  rules  possible 
were  generated  for  this  one  portion  using  the  knowledge 
collected  with  each  of  the  three  knowledge  acquisition 
techniques.  As  the  rules  were  formulated,  the  number  of 
inferences  required  to  complete  a  rule  were  noted.  Unlike 
the  previous  test,  the  knowledge  collected  by  all  three 
techniques  was  evaluated  using  the  same  portion  of  the  work 
breakdown  process.  Table  3  illustrates  the  results  from 
this  test. 

Interruption  analysis  provided  the  greatest  number  of 
rules  for  this  portion  of  the  work  breakdown  structure. 
Concept  mapping  and  interviewing  provided  virtually  the  same 
number  of  rules.  Of  course,  this  analysis  does  not  imply 
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Table  3. 

Number  of  Rules  Formulated 
and  Inferences  Required  to  Formulate  Rules 


Number  of  Rules 

Number  of  Inferences 

Interview 

15 

7 

Concept 

16 

1 

Napping 

Interrupt  ion 

20 

13 

Analysis 

that  with  subsequent  interviews  or  concept  mapping  sessions, 
additional  rules  could  not  be  generated.  Given  the  fixed 
and  somewhat  limited  time,  interruption  analysis  proved 
superior  for  rule  generation. 

Concept  mapping  required  the  fewest  inferences  to 
complete  its  set  of  rules.  Interruption  analysis  required 
the  greatest  number  of  inferences  because  there  was  little 
explanation  provided  for  the  actions  that  the  expert  took 
unless  something  was  done  that  the  investigator  clearly  did 
not  understand.  Otherwise,  the  investigator  assumed  certain 
rationales  for  certain  actions.  As  a  consequence,  the 
knowledge  gained  by  this  technique  was  implied  by  the 
expert's  actions  rather  than  verbally  explained.  The 
results  from  this  test  reflect  the  fact  the  investigator 
made  assumptions  about  rules  for  instances  when  he  did  not 
ask  the  expert  the  exact  reason  for  the  expert's  action. 
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Pi f ferences  in  Knowl edge  Col lected.  When  acquiring 
expert  knowledge,  it  is  important  to  access  the  breadth  and 
depth  of  an  expert's  knowledge  about  a  given  task.  This 
test  attempted  to  determine  if  any  of  the  techniques  missed 
whole  blocks  of  knowledge  important  to  developing  a  work 
breakdown  structure.  The  next  test  is  a  measure  of  the 
differences  in  the  knowledge  collected  by  the  three 
knowledge  acquisition  techniques.  For  the  first  test,  the 
composite  knowledge  collected  by  each  technique  was  divided 
into  four  general  areas.  "Background  information"  relates 
to  information  about  the  work  breakdown  structure.  This 
includes  the  purpose  of  the  work  breakdown  structure, 
organizations  within  a  program  office  that  may  use  a  work 
breakdown  structure,  and  some  of  the  terminology  associated 
with  the  work  breakdown  structure.  "WBS  Format"  refers  to 
the  accepted  format  of  the  printed  work  breakdown  structure 
for  any  particular  project  or  program.  "Differences  in 
WBSs"  addresses  the  differences  between  the  production  and 
development  work  breakdown  structures.  Lastly,  "WBS 
Elements"  refers  to  knowledge  about  individual  elements  of 
the  work  breakdown  structure  at  either  the  task  (level  2)  or 
sub-task  (level  3)  level. 

As  Table  4  depicts,  the  interview  method  addressed  each 
of  these  areas.  Concept  mapping  failed  to  address  proper 
formatting  of  a  work  breakdown  structure,  and  interruption 
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analysis  provided  no  background  information  about  the  work 


breakdown  structure. 


Table  4. 

Overview  of  the  Knowledge 
Collected  Using  Each  Technique 


Background 
Informat  ion 

UBS  Format 

Di f  ferences 
in  UBSs 

UBS  Elements 

Interview 

YES 

YES 

YES 

YES 

Concept  Mapping 

YES 

NO 

YES 

YES 

Interrupt  ion 
Analysis 

NO 

YES 

YES 

YES 

Next,  a  prototype  expert  system  was  developed.  The 
knowledge  collected  using  all  three  techniques  was 
integrated  into  a  complete  knowledge  base  of  the  work 
breakdown  structure  process.  This  aggregate  knowledge  base 
was  then  used  to  create  an  expert  system.  Subsequent 
measures  of  the  relative  effectiveness  of  the  three 
knowledge  acquisition  techniques  were  done  using  this 
prototype  expert  system.  The  code  for  this  expert  system 
car  be  found  in  Appendix  B. 

The  second  measure  of  the  differences  in  the  knowledge 
collected  was  to  assess  the  contribution  each  technique  made 
to  the  aggregate  expert  system.  This  was  done  after  the 
prototype  expert  system  was  complete  and  before  the  system 
was  val idated. 
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Each  rule  of  the  expert  system  was  evaluated  to 
determine  if  it  could  have  been  created  from  the  knowledge 
collected  from  each  technique.  The  expert  system's  seventy- 
nine  rules  were  divided  into  the  general  areas  addressing 
hardware,  other  contract  costs  (OCC),  and  other  government 
costs  (OGC).  Each  of  these  areas  was  further  divided  into 
rules  addressing  either  the  project  level  (Level  1),  task 
level  (level  2)  or  the  sub-task  level  (level  3).  Table  5 
illustrates  the  results  of  this  test. 


Table  5. 

Number  of  Rules  Contributed  to  the  Aggregate 
Expert  System  By  Each  Knowledge  Acquisition  Technique 


Interview  !  Concept  Map  !  Int.  Analysis 


Hardware 

Level  1 

Rule  10 
Rule  20 


sub-total 

0 

Level  2 

Rule 

30 

X 

Rule 

40 

Rule 

240 

sub -total 

1 

Level  3 

Rule 

50 

Rule 

60 

Rule 

70 

Rule 

80 

X 

Rule 

90 

Rule 

100 

Rule 

110 

Rule 

120 

Rule 

130 

Rule 

140 

Rule 

150 

Rule 

160 

X 

X 

X 

X  X 

X 
X 
X 
X 
X 
X 
X 
X 
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Table  5  continued 


Interview  !  Concept  Map  !  Int.  Analysis 


Rule  170 

X 

X 

X 

Rule  180 

X 

Rule  190 

X 

Rule  200 

X 

Rule  210 

X 

Rule  220 

X 

Rule  230 

X 

Rule  250 

X 

Rule  260 

X 

Rule  270 

X 

Rule  280 

X 

Rule  290 

X 

Rule  300 

X 

Rule  310 

X 

X 

Rule  320 

X 

X 

Rule  330 

X 

X 

total 

5 

5 

25 

Other  Contract  Costs 


Level  2 

Rule  340  X 
Rule  350  X 
Rule  360  X 
Rule  410  X 
Rule  420  X 
Rule  460  X 
Rule  510  X 
Rule  520  X 
Rule  530  X 
Rule  540  X 
Rule  570  X 
Rule  600  X 
Rule  610 

Rule  620  X 
Rule  630  X 
Rule  640  X 
Rul e  650  X 
Rule  660  X 
Rule  670  X 
Rule  680  X 


X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 

X 


sub -total 


19  18  8 


Level  3 

Rule  370 
Rule  380 
Rule  390 


35 


X 

X 

X 


Table  5  continued 


Interview  ! 

!  Concept  Hap  ! 

!  Int.  Analysis 

Rule  400 

X 

Rule  430 

X 

Rule  440 

X 

Rule  450 

X 

Rule  470 

X 

X 

Rule  480 

X 

X 

Rule  490 

X 

Rule  500 

X 

X 

Rule  550 

X 

X 

Rule  560 

X 

X 

Rule  580 

X 

Rule  590 

X 

sub-total 

0 

5 

15 

Other  Government  Costs 

Level  2 

Rule  690 

X 

X 

Rule  700 

X 

X 

Rule  710 

X 

X 

Rule  720 

X 

X 

Rule  730 

X 

X 

Rule  740 

X 

X 

Rule  750 

X 

X 

Rule  760 

X 

X 

Rule  770 

X 

Rule  780 

X 

X 

Rule  790 

X 

X 

X 

sub-total 

10 

9 

1 

total 

35 

38 

54 

Clearly,  interruption  analysis  provided  the  greatest 
number  of  rules  overall  (68*/.  of  the  total)  and  for  the  more 
detailed  level  3  work  breakdown  structure  elements. 
Interviewing  and  concept  mapping  provided  results  similar  to 
each  other  (44*/.  for  the  interview  and  4B7.  for  concept 
mapping),  focusing  on  the  task  or  level  2  work  breakdown 
structure  elements  associated  with  other  contract  costs  and 
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other  government  costs.  The  reader  is  reminded  that  these 

rules  may  have  required  inferences  to  complete  the 

formulation  of  the  rule.  As  the  reader  will  recall  from  the 

"Rule  Formulation  and  Inferences  Required"  portion  of  this 

analysis,  interruption  analysis  required  inferences  when 

formulating  65 %  of  its  rules,  interviews  required  inferences 

for  47%  of  its  rules,  and  concept  mapping  required 

inferences  for  6%  of  its  rules. 

Number  of  Knowl edge  Trans for mat  ions.  The  more  changes 

required  to  produce  an  expert  system  rule,  the  more  time  it 

will  take  the  knowledge  engineer  to  create  an  expert  system 

rule  base,  and  the  greater  the  possibility  for 

interpretat ion  errors  or  errors  of  omission.  Thus,  fewer 

transf ormat ions  of  knowledge  would  be  preferred.  The 

transcribing  of  knowledge  has  been  discussed  in  the  section 

entitled  "Technique  Implementation."  The  number  of 

knowledge  transformat  ions,  however,  refers  to  the  number  of 

changes  the  knowledge  must  undergo  to  become  an  expert 

system  rule  with  proper  syntax. 

For  example,  detailed  notes  from  an  interview  audiotape 

may  have  yielded  the  following: 

. . . MIL-STD-B81  does  not  address  differences 
between  a  production  and  a  development  work 
breakdown  structure.  For  instance,  software  and 
software  integration  are  common  work  breakdown 
structure  elements,  but  software  development 
should  be  completed  during  program  development. 

As  such,  no  software  elements  should  appear  in  the 
production  work  breakdown  structure. 
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These  notes  represent  one  transformat  ion  of  the  knowledge: 

tape  to  notes.  The  next  transformat  ion  would  create  the 

production  rule  from  the  notes.  A  production  rule  puts  the 

knowledge  in  the  following  form: 

IF:  This  is  a  development  work  breakdown 

structure,  and  computer  programs  are  used  by  the 
program  hardware 

THEN:  Hardware/Sof t ware  Integration  should  be 
included  as  a  level  3  work  breakdown  structure 
element  under  Hardware. 

This  production  rule  represents  the  second  transformat  ion 
for  the  expert's  knowledge.  The  third  and  final 
transf ormat ion  of  the  knowledge  would  be  the  translation  of 
the  production  rule  into  the  proper  expert  system  syntax: 

IF:  WBS_type=devel opment  AND 

computer _programs=yes 

THEN:  1  evel  _3=Hardware_Sof tware_Integrat ion 
The  steps  involved  in  preparing  an  expert  system  rule 
for  the  interview  included  transferring  the  knowledge  from 
the  audiotape  to  a  written  format;  producing  a  production 
rule  from  the  written  text;  then  producing  an  expert  system 
rule. 

Concept  mapping  involved  the  same  steps.  However, 
instead  of  detailed  notes  of  the  audiotapes,  concept  maps 
were  produced.  Production  rules  were  then  created  from  the 
concept  maps,  and  finally,  an  expert  system  rule  was 
encoded . 
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Interruption  analysis  involved  the  fewest  number  of 
knowledge  transformat  ions.  Since  verbal  information  was 
exchanged  only  when  the  investigator  did  not  understand  an 
action,  the  audiotapes  of  these  sessions  provided  no  really 
new  information.  The  investigator  was  able  to  collect  all 
of  the  necessary  information  in  written  form,  even  when 
questions  were  asked.  For  interruption  analysis, 
transformat  ion  of  data  consisted  of  forming  production  rules 
from  written  notes  and  then  translating  the  production  rules 
into  the  proper  syntax  for  the  expert  system  software. 

Table  6  presents  a  summary  of  the  numerical  data  from  this 
test . 


Table  6. 

Number  of  Knowledge  Transformat  ions 
Necessary  to  Create  an  Expert  System  Rule 


Concept 

Interrupt  ion 

Interview 

Mapping 

Analysis 

Transformat  ions 
Required 

3 

3 

2 

Expert  System 

Val idat ion. 

The  most 

important  test  of 

any  knowledge  acquisition  technique  is  how  well  the 
resulting  expert  system  reflects  the  expert's  knowledge. 
The  technique  that  causes  the  fewest  knowledge  base  errors 
would  be  the  better  knowledge  acquisition  technique.  As  a 
final  measure  of  these  knowledge  acquisition  techniques,  a 
limited  prototype  expert  system  was  developed  integrating 
the  information  collected  from  all  three  knowledge 
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acquisition  techniques.  This  expert  system  was  then  taken 
back  to  the  part ic ipat ing  experts  to  determine  if  these 
techniques  could  aid  the  development  of  an  expert  system 
that  accurately  produced  a  work  breakdown  structure. 

To  validate  the  expert  system,  the  experts  were  asked 
to  construct  an  appropriate  work  breakdown  structure  using  a 
fictional  case  study.  The  work  breakdown  structure  was 
constructed  using  the  prototype  expert  system.  The  expert's 
were  asked  to  carefully  analyze  the  system's  output  as  well 
as  the  decision  rules  of  the  expert  system. 

Discrepancies  were  noted  and  then  analyzed  to  determine 
the  nature  of  the  errors.  Generally,  errors  were  assumed  to 
be  the  result  of  incomplete  information  resulting  in  faulty 
inferences,  errors  in  the  actual  knowledge  conveyed  to  the 
expert  system  programmer,  or  errors  that  resulted  from 
mistakes  in  expert  system  programming. 

The  first  expert,  who  was  the  subject  for  the 
interviews,  identified  six  discrepancies  in  the  expert 
system  execution.  All  of  the  discrepancies  related  to 
differences  between  the  development  and  production  work 
breakdown  structures.  Five  of  these  discrepancies  resulted 
from  insufficient  information  received  during  the  knowledge 
acquisition  sessions.  In  each  case,  the  decision  rule  was 
correct  for  the  development  work  breakdown  structure,  but 
incorrect  for  the  production  work  breakdown  structure.  Of 
these  five  incorrect  rules,  four  of  the  rules  had  been 
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addressed  during  the  interview  sessions  and  one  had  not. 

Even  though  information  pertaining  to  the  four  rules  had 
been  provided  by  the  expert,  this  information  was  not 
sufficiently  clear  to  prevent  an  error  in  its  encoding.  The 
other  inference  error  resulted  when  the  investigator 
encountered  a  specific  work  breakdown  structure  element  that 
had  not  been  addressed  during  the  interview  sessions.  These 

five  errors  were  determined  to  be  attributable  to  the 
knowledge  acquisition  technique. 

In  the  remaining  case,  the  expert  felt  that  instead  of 
automatically  including  a  particular  work  breakdown 
structure  element,  a  question  requiring  user  input  should  be 
used  to  decide  if  the  element  should  be  included.  This 
discrepancy  was  the  result  of  conflicting  information 
received  by  the  investigator  from  the  experts. 

The  second  expert,  who  was  the  subject  for  the  concept 
mapping  sessions,  noted  only  two  discrepanc ies.  One  error 
was  the  same  as  that  identified  by  the  first  expert  and 
involved  a  difference  between  a  development  and  a  production 
work  breakdown  structure.  As  with  the  first  expert,  this 
error  was  attributed  to  an  incorrect  inference  on  the  part 
of  the  investigator  and  Judged  to  be  attributable  to  the 
knowledge  acquisition  technique.  This  area  had  also  been 
addressed  during  the  initial  concept  mapping  sessions.  The 
second  error  involved  a  difference  in  the  knowledge  between 
two  of  the  experts.  One  expert  who  had  contributed  to  the 
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knowledge  base  felt  a  particular  element  of  the  work 
breakdown  structure  was  distinct,  yet  this  expert  felt  that 
the  element  was  not  distinct,  merely  synonymous  with  a 
second  work  breakdown  structure  element  and  should  not  be 
included  as  an  element  in  a  work  breakdown  structure. 

The  third  expert,  who  was  the  subject  for  the 
interruption  analysis  technique,  identified  three 
discrepanc ies.  One  discrepancy  dealt  with  which  particular 
level  a  certain  work  breakdown  structure  element  should  be 
found  in.  This  discrepancy  was  attributed  to  a  difference 
in  the  expert's  knowledge.  This  expert  also  noted  that  the 
work  breakdown  structure  element  "data  processing  equipment" 
appeared  only  in  the  production  version  of  the  work 
breakdown  structure  when  it  should  have  appeared  in  the 
development  version  also.  Upon  investigation,  it  was 
discovered  that  the  expert  system  had  prompted  the  expert 
about  this  element  for  the  development  work  breakdown 
structure,  but  the  expert  had  neglected  to  include  this 
element  when  asked  the  first  time.  The  expert  system  was 
not  at  fault.  On  another  topic,  this  expert  also  felt  that 
one  rule  regarding  missile  system  propulsion  should  be 
added.  This  last  error  was  determined  to  be  a  programming 
error  on  the  part  of  the  investigator.  The  method  that  the 
investigator  used  to  program  this  particular  section  allowed 
a  correct  element  to  be  included,  but  the  expert  felt  the 
decision  rule  was  not  adequate  to  elicit  a  proper  response 
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from  a  system  user.  Neither  of  the  other  two  experts  noted 
the  problem. 

General  Observat ions 

Technique  I mp 1 ement at i on .  The  numerical  data  does  not 
reflect  all  the  pertinent  information  relative  to  technique 
implementation.  There  are  a  number  of  general  observations 
that  are  important. 

First,  this  researcher  felt  that  interviews  were  the 
easiest  technique  to  use,  followed  by  interruption  analysis, 
and  lastly  concept  mapping.  This  was  due  to  the  fact  that 
the  interview  required  little  preparation,  and  transcr ipt ion 
of  the  interview  tapes  was  not  complicated.  If  the  entire 
interview  had  been  transcribed  "word-for-word, "  the 
interview  would  have  been  considerably  more  difficult  than 
either  of  the  other  two  techniques. 

Second,  the  expert  who  participated  in  the  concept 
mapping  technique  was  not  comfortable  with  this  format.  The 
expert  experienced  a  great  deal  of  difficulty  with  the 
propos 1 t ional  linkages,  and  trying  to  find  an  appropriate 
proposition  proved  time  consuming  and  distracting.  To  help 
the  flow  of  information,  and  put  the  expert  more  at  ease, 
the  concept  maps  were  de-emphasized  during  the  sessions. 
Smaller  concept  maps  were  drawn  or  simple  lists  were 
generated.  This  researcher  then  integrated  these  smaller 
maps  and  lists  into  a  larger  map  that  provided  the  basis  for 
the  following  sessions. 
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Lastly,  if  it  were  not  for  having  to  develop  case 
studies  for  the  interruption  analysis,  this  technique  would 
have  been  the  easiest  to  implement.  Watching  the  expert 
perform  a  task  was  all  the  catalyst  necessary  to  generate 
questions  and  convey  information. 

Rule  and  Fact  Product  ion.  The  numerical  data  presents 
a  fairly  accurate  picture  of  the  relative  effectiveness  of 
the  three  techniques  in  rule  and  fact  production.  The 
graphical  representat ion  of  knowledge  as  provided  by  the 
concept  mapping  technique  proved  the  easiest  method  to  use 
for  drafting  production  rules.  Appendix  C  represents  the 
concept  map  that  resulted  from  the  three  knowledge 
acquisition  sessions.  Not  only  were  the  relationships  clear 
and  concise,  but  the  concept  map  required  the  least  amount 
of  time  to  search  and  find  all  of  the  pertinent 
rel at ionships. 

The  most  difficult  knowledge  1c  work  with  was  that  of 
the  interview.  This  data  was  only  loosely  structured,  and 
as  such  required  a  good  deal  of  effort  to  identify  pertinent 
information  and  then  deduce  important  rel at ionships  within 
that  information. 

While  the  knowledge  collected  using  interruption 
analysis  provided  the  greatest  number  of  production  rules, 
it  also  required  the  greatest  number  of  inferences.  Even 
though  this  knowledge  was  very  process  oriented,  it  suffered 
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some  of  the  same  disorganizat ion  as  the  information 
collected  from  the  interview. 

Even  though  interruption  analysis  required  the  greatest 
number  of  inferences,  validation  of  the  expert  system 
indicated  that  interviews  allowed  the  greater  number  of 
inaccuracies  in  the  expert  system  rule  base.  While  the 
expert  system  represented  a  composite  of  all  the  knowledge 
collected,  the  errors  identified  by  the  interviewed  expert, 
with  one  exception,  were  specific  to  knowledge  collected 
during  the  interview  sessions. 
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V.  Recommendat ions 

Research  Review 

Knowledge  acquisition  presents  a  challenging  aspect  of 
expert  system  development  for  the  knowledge  engineer. 

Review  of  existing  expert  system  and  artificial  intelligence 
literature  provided  little  useful  information  about  deciding 
which  knowledge  acquisition  technique  or  techniques  would  be 
the  most  effective  for  a  given  task.  It  was  the  purpose  of 
this  research  project  to  evaluate  the  relative  effectiveness 
of  three  knowledge  acquisition  techniques  for  developing  an 
expert  system  to  produce  a  project  work  breakdown  structure. 

Cond i t ions  and  Recommendat ions 

There  are  a  number  of  conclusions  that  can  be  drawn 
f  om  this  study.  Perhaps  the  most  obvious  conclusion  is 
that  no  one  technique  is  clearly  superior  for  developing  a 
work  breakdown  structure  expert  system  under  all  the 
conditions  that  a  knowledge  engineer  may  encounter. 

However,  if  the  knowledge  engineer  is  faced  with  certain 
conditions  surrounding  the  project,  then  a  distinct  order  of 
effectiveness  among  the  techniques  becomes  evident. 

Before  specific  conditions  and  recommendations  are 
provided,  the  reader  should  remember  the  followings 
-The  results  of  this  experiment  pertain  only  to 
developing  an  expert  system  to  produce  a  project  work 
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breakdown  structure  and  may  not  be  appropriate  for  other 
tasks. 

-One  of  the  techniques  evaluated  in  this  experiment, 
interruption  analysis,  requires  a  task  that  can  be  observed. 

For  the  reader's  convenience,  Table  7  provides  a  review 
of  the  data  collected  during  this  research.  Table  7 
provides  the  totals  or  averages  for  each  of  the  quantitative 
measures  used  to  evaluate  each  knowledge  acquisition 
technique's  relative  effectiveness. 

The  recommendations  of  this  research  are  the  result  of 
evaluating  each  technique  according  to  three  general 
character ist ics.  The  first  char ac t er i st ic  is  that  of  the 
quality  of  the  knowledge  extracted.  Quality  includes  the 
accuracy  and  completeness  of  the  knowledge.  The  second 
character ist ic  is  that  of  the  time  required  to  exercise  each 
technique.  Project  time  constraints  may  impose  limitations 
on  either  the  knowledge  engineer,  the  expert,  both,  or 
neither  individual.  The  third  character ist ic  is  the 
quantity  of  the  knowledge  collected.  Quantity  includes  the 
breadth  of  the  knowledge  elicited  and  the  number  of  expert 
system  rules  created.  Each  of  the  following  recommendations 
is  preceded  by  conditions  that  the  knowledge  engineer  may 
face  when  acquiring  expert  knowledge. 
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Table  7. 

Summary  of  Research  Data 


Test 

Interview 

Concept 

Happing 

Interrupt  ion 
Analysis 

Implement at  ion 

Times  (totals) 

419.1  m 

630*  0  m 

505.4  m 

Rule  Formulat ion 

Times  (average) 

4.4  m 

2  m  8  m 

6.0  m 

Number  of  Rules 
(for  a  specific  part 
of  the  expert  system) 

15 

16 

20 

Number  of  Inferences 
Required  to  Encode 
Previous  Rules 

7 

1 

13 

Areas  of  Incomplete 
Knowledge 

none 

UBS  format 

Background 

Number  of  Rules 
Contributed  to  the 

Expert  System 

35 

3B 

54 

Number  of  Knowledge 
Transformat  ions 

3 

3 

2 

Number  of  Errors  Created 
in  the  Expert  System 
Knowledge  Base 

5 

1 

0 

Cond i  t  ions  and  Recommendat  ion  1_: 

IF  no  time  constraints  are  imposed  on  either 
the  knowledge  engineer  or  the  project  expert 
(optimal  conditions) — 

AND  IF  a  large  number  of  rules  is  a  high 
pr ior ity-- 

AND  IF  the  accuracy  of  the  knowledge  is  a 
high  priority-- 

AND  IF  breadth  of  knowledge  is  a  high 
pr ior i ty — 
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THEN  a  combination  of  concept  mapping  and 
interruption  analysis  would  be  the 
acquisition  technique  of  choice. 

This  combination  of  techniques  provided  the  broadest 
range  of  knowledge,  the  most  expert  system  rules,  and  the 
fewest  expert  system  errors.  Since  few  knowledge  engineers 
encounter  optimal  conditions,  especially  in  the  areas  of 
their  own  time  or  the  time  of  the  experts  whose  knowledge 
they  need,  other  acquisition  techniques  or  combinations  of 
techniques  may  be  needed,  as  described  below.  Nonetheless, 
if  the  situation  permits  the  combination  of  concept  mapping 
and  interruption  analysis,  the  results  should  be  a  large 
number  of  accurate  rules. 

Condit ions  and  Recommendat ion  2: 

IF  time  is  constrained  for  either  the 
knowledge  engineer  or  the  project  expert — 

AND  IF  a  large  number  of  expert  system  rules 
is  a  high  priority — 

AND  IF  the  accuracy  of  the  knowledge  is  a 
high  priority — 

AND  IF  breadth  of  knowledge  is  a  low 
prior ity — 

THEN  interruption  analysis  would  be  the 
acquisition  technique  of  choice. 

Interruption  analysis  provided  the  most  expert  system 
rules  and  fewest  expert  system  errors.  In  addition, 
interruption  analysis  did  not  require  the  greatest  amount  of 
impl ementat ion  time.  Although  encoding  expert  system  rules 
from  the  interruption  analysis  notes  takes  slightly  longer 
than  is  necessary  when  encoding  from  interview  notes  or 
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concept  maps,  this  liability  is  offset  by  the  fact  that 
interruption  analysis  produced  42*/.  more  expert  system  rules 
than  the  next  best  technique. 

Cond i t ions  and  Recommendat ion  3: 

IF  the  knowledge  engineer  has  a  limited 
understanding  of  the  task  for  which  the 
expert  system  is  being  developed — 

AND  IF  the  accuracy  of  the  knowledge  is  a 
high  priority — 

AND  IF  the  knowledge  engineer  has  only 
limited  time  to  spend  with  the  project 
expert — 

THEN  concept  mapping  would  be  the  acquisition 
technique  of  choice. 

Concept  mapping  provided  sufficient  breadth  of 
knowledge,  more  expert  system  rules  than  the  interview, 
fewer  expert  system  errors  than  the  interview,  and  quicker 
rule  encoding  than  either  interviews  or  interruption 
analysis.  Concept  mapping  also  required  the  least  amount  of 
interaction  time  with  the  expert. 

Cond it  ions  and  Recommendat ion  At 

IF  th  e  are  overall  project  time 
constraints— 

AND  IF  the  knowledge  engineer  does  not  have  a 
great  deal  of  time  to  prepare  for  the 
knowledge  acquisition  sessions — 

AND  IF  the  accuracy  of  the  rules  is  not  a 
high  priority — 

THEN  interviews  would  be  the  acquisition 
technique  of  choice. 

Interviews  required  the  least  preparation  time  and  the 
1  east  overall  implementation  time,  and  provided  a  number  of 
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rules  comparable  to  concept  mapping.  However,  interviews 
also  created  the  most  expert  system  errors.  While 
interviews  do  have  limitations,  interviews  can  be  useful  in 
developing  a  prototype  expert  system  to  test  the  feasibility 
of  the  project. 

Recommendat ions  for  F urther  Research 

A  great  deal  of  additional  research  remains  to  be  done 
in  the  areas  of  knowledge  acquisition  and  the  application  of 
expert  system  technology.  In  general ,  there  are  at  least 
three  areas  with  the  potential  for  follow-on  research  to 
this  project. 

First,  there  are  a  number  of  other  knowledge 
acquisition  techniques  that  this  investigator  did  not 
evaluate.  Additional  experiments  similar  to  this  one, 
examining  different  combinations  of  knowledge  acquisition 
techniques,  could  be  used  to  evaluate  the  relative 
effectiveness  of  other  techniques  such  as  repertory  grid 
analysis,  protocol  analysis,  hierarchical  clustering, 
questionnaires,  and  multidimensional  scaling.  These 
experiments  may  identify  knowledge  acquisition  techniques  or 
combinations  of  techniques  that  are  more  effective  than 
those  evaluated  by  this  research. 

Second,  the  same  three  knowledge  acquisition  techniques 
evaluated  by  this  research  could  be  tested  using  an 
experiment  similar  to  this  one  but  with  a  different  problem 
domain  and  task  to  see  if  similar  results  could  be  obtained. 
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An  experiment  or  series  of  experiments  may  identify  one  or  a 
specific  combination  of  these  knowledge  acquisition 
techniques  as  being  generally  more  effective  across  a  broad 
assortment  of  problem  domains,  tasks,  and  project 
condit ions. 

Third,  the  reader  is  reminded  that  a  prototype  expert 
system  was  developed  during  the  course  of  this  research. 
While  this  expert  system  was  limited  in  its  capability,  it 
does  provide  a  starting  point  from  which  a  more  functional 
expert  system  could  be  developed. 

Summary 

As  stated  earlier,  the  purpose  of  this  study  was  to 
evaluate  the  relative  effectiveness  of  the  three  knowledge 
acquisition  techniques;  interview,  concept  mapping,  and 
interruption  analysis.  While  no  empirical  evidence  existed 
in  the  1 iterature  to  assist  this  researcher  in  his  choice  of 
appropriate  knowledge  acquisition  techniques  for  this 
experiment,  the  three  techniques  selected  proved  to  be 
useful  for  developing  a  work  breakdown  structure  expert 
system.  Quantitative  measures  were  developed  to  assess  the 
relative  effectiveness  of  these  three  knowledge  acquisition 
techniques.  These  measures  addressed  such  issues  as  the 
implementation  of  the  technique,  the  analysis  of  the 
resulting  knowledge,  the  contribution  each  technique  made  to 
the  final  expert  system,  and  the  number  of  errors  in  the 
final  expert  system  attributable  to  the  knowledge 
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acquisition  technique.  The  results  of  this  experiment 
indicated  that  under  optimal  expert  system  development 
conditions,  a  combination  of  techniques  employing  concept 
mapping  and  interruption  analysis  would  be  the  most 
effective  choice.  While  no  one  technique  was  identified  as 
being  clearly  superior  for  all  the  possible  conditions  a 
knowledge  engineer  may  encounter,  the  strength  of  this 
experiment  is  that  it  empirically  identifies  important 
strengths  and  weaknesses  of  each  knowledge  acquisition 
technique,  and  conditions  where  each  technique  or  a  specific 
combination  of  techniques  is  a  better  choice  than  others. 
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Appendix  A:  Sampl  e  of  a_  "Word-for-Word" 

Transcr jpt ion  of  an  Interview 

The  following  is  a  sample  "word-for-word"  transcr ipt ion 
of  a  portion  of  one  interview  session.  It  represents  about 
9.6  minutes  of  interview  content,  and  is  intended  to 
illustrate  the  effort  required  to  transcribe  an  interview. 
This  transcript  took  about  69  minutes  to  transcribe.  This 
is  approx imatel y  a  7  to  1  ratio  of  transcr ipt ion  time  to 
actual  interview  tape  content. 


Investigator:  Let's  talk  about  the  WBS  process.  And  to 

that  extent,  I'm  going  to  let  it  free-wheel.  Basically, 
we'll  treat  this  as  a  student-teacher  relationship.  I  want 
you  to... to  teach  me  the  process  of  developing  a  WBS  in 
terms  of  RW's  use.  In  as  general  terms  as  you  can  for  right 
now.  I  will  go  home  and  analyze  the  data  and  generate  more 
specific  questions. 

Expert:  Ah.... Of  course  MIL-STD-BB1  is  where  it  starts. 

So,  BBl  should  be  in  the  background  of  everybody's  thinking 
as  they  are  doing  this.  Ah...ACCM's  desire  or  what  they  are 
willing  to  sign  off  on  should  be  the  next  screen  we 
consider.  ACCM  is. . . is  primarily  concerned  about  getting  us 
to... develop  and  get  on  contract  WBS's  that  will  produce 
data  that  will  reasonably  easily  fit  into  OSD  CAIG  and  A F 
CAIG  data  bases.  Ah... the  next  thing  we  have  to  take  into 
consideration  is  the  real  program.  Many  times... ah,  when  we 
do  WBSs  here,  we  are  doing  it  with  junior  level  analysts  and 
many  times  junior  level  program  managers  and  a  fair  amount 
of  time... ah,  occasionally  I'd  say,  put  it  that  way,  junior 
level  engineers.  So  we  are  usually  dealing  with  a  fairly 
green  team. 

When  we  start  to  develop  a  WBS,... the  analysts,  the 
cost  analyst,  usually  takes  the  lead... ah,  at  least  from  my 
per spect ive. .. usual  1 y  takes  the  lead  because  we  deal  with 
WBS  on  many  different  programs  so  we  have  a  better  feel  for 
them  across  the  board  than  the  program  mi  ager  who  deals 
with  it ...  perhaps,  once  in  a  lifetime  or  maybe  twice. 

Now  here  again,  guide  me  if  I'm  not  going  down  the 
right  path. 
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We  normally  look  at  3  major  categories  of  stuff.  Well, 
let  me  back  off  for  a  second...  The  program  we  are 
concerned  f irst ...  about  development,  production  and  OfcS. 
Okay, .... 

Invest igator :  And  OfcS  is? 

Expert:  Operation  and  support.  So,... ah,  normally  when  we 

go  on  contract,  we  are  going  on  for  some  or  all  of  this. 

The  instant  contract  is  usually  for  development  effort,  and 
may be  with  some  priced  options.  Rarely  do  we  have  any  O&S 
on  contract  yet.  From  ah,  an  ACCM  perspective,  they  want  to 
see  a  WBS...like  that...  And  then  they  sa. . . ,  you  know. .. In 
other  words,  they  are  concerned  with  the  development 
program,  and  then  later  they  are  concerned  with  the 

production  program.  Okay,  in  talking  with  _ ,  he 

said  when  you  come  back  with  a... when  you  get  ready  to  have 
another  major  contractual  action  you  need  to  bring  him 
another  WBS  for  that  effort,  or  show  him  how  the  one  he 
previously  approved  has  been  revised.  Ah,  in  talking  off 
the  record,  I  said  gee,  I've  never  worked  a  program  that 
ever  brought  another  WBS  back  through  you.  He  said  well,  he 
can't  be  the  policeman  for  all  of  ASD,  he  says  they  are 
supposed  to.  He  says  rarely  do  they,  but  they  are  supposed 
to. 


Ah,  development  though,  breaks  down  into  prime  mission 
equipment,  OCC-other  contractor  costs,  and  other  government 
costs.  So  we  know  that  there  is  a  family  of  things  under 
each  of  those,  but  this  is  really  the  big  question  mark. 

Invest igator :  The  prime  mission  equipment? 

Expert:  Yes.  The  PME  itself  breakdown  typically  in  one  of 
two  ways.  It  either  breaks  down  directly  into  what  we  call 
hardware,  software,  or  hardware/so f t ware  integration 
effort... or  it  will  breakdown  into... a  major  segment  or 
segments  with  the  integration  of  those  segments  together. 

And  each  of  those  having  hardware,  software,  and  stand-alone 
integration  and  again  hardware,  sof tware. . . I ' m  going  to  give 
you  these  drawings  so  you  can  relate  to  them  later. 

Okay,  from  _ 's  perspective,  I'll  keep  going 

back  to  that  because  he  is  our  local  interpreter  of  MIL-STD- 
881.  He  wants  to  see  the. .. equipment  at  a  reasonably  high 
level.  He  wants  to  avoid  breaking  the  details  down  to  low. 
(Jm... his  contention  is  on  that  one  that... the  lower  level  we 
spec i fy. .. that ' s  the  lower  Ip  ’1  that  the  contractor  needs 
to  issue  work  author izat ion  p^-kages  out  to... out  to  his, 
what?. . . funct ional  representat i ves.  Okay,  I  guess  they  got 
to  write  up... ah,  sell  orders  or  something  like  that.  Each 
contractor  calls  it  something  different. 
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Okay,  going  down  this  path,  the  hardware  will  generally 
breakdown  into  specific  line  replaceable  units  or  what  we 
like  to  consider ... roughl y  line  replaceable  units.  In  some 
cases  we  have  wierd  stuff  and  it  doesn't  really  breakdown 
that  way.  Those  are  the  black  boxes  we  are  trying  to 
estimate.  And,  software  breaks  down  into  CPC Is,  or  computer 
program  configuration  items. 

By  putting  what  _ ,  what  ACCM  calls  a  false 

summing  level  here,  we  force  these  down  another  level. 

Okay,  so  if  _  says  here  that  this  is  the  development 

program. .. which  is  different  from  that  program,  this  is  the 
development  program  is  the  only  WBS  he  wants  to  look  at . . . at 
one  time.  If  the  development  program  which  is  level  one. 
That's  a  confusion  factor  that... ACCM  is  locked  on  real 
tight.  Ah,  my  recommendation  is  because  of  what... of  what 
can  appear  at  one  level  depending  upon  how  you've  drawn  the 
WBS,  the  exact  same  LRU  can  be  at  level  3,  4,  or  5.  Just 
based  on  how  many  false  summing  levels  you  put  in  there.  If 
this  program  is  level  1,  level  2,  level  3,  that  gets  the 
LRU's  at  level  4....  Okay,  i f  we  get  rid  of  this,  move  all 
this  stuff  up,  then  they  have  moved  up  to  level  3  where 

_  would  really  like  to  see  them.  Which  means  that 

on  that  path,  we  really  have,  there  is  PME,  Just  replicating 
that  PME  level  there.  You  have  hardware  item,  hardware, 
hardware,  hardware,  integration  of  the  hardware.  Software, 
software,  software,  integration  of  the  software.  And  then 
hardware-sof tware  integration.  And  that  puts  it  back  up 
at... the  LRUs,  or  the  ah,  lowest  level  we  go  to  is  now  at 
level  3.  That's  truly  a  confusion  factor.  This  is  the 
direction  I'm  trying  to  drive  because  it  is  what  ACCM  is 
looking  for,  and  more  closely  replicates  BB1. 

Investigator:  That's  what  I've  heard  reference  to  as  dummy 

levels  basically? 

Expert:  Yes... Dummy  levels  or  false  summing  levels.  They 

are  real  handy  you  know. .. for  us.  And  ah,  something  that  is 
important  is  that  ah. .. f inane ial  managers  here  at  RW  and  a 
couple  other  SPO's  are  the  ones  who  really  put  out  the 
official  WBS.  And  they  do  that  via  a  form  called  the 
Systems  Command  SCIPS.  AFSC  Form  126.  We're  developing  a 
WBS  for  our  purposes,  and  that's  to  do  a  cost  estimate. 

They  are  the  ones  who  should  be  doing  a  WBS  that  gets  it 

through  the  _ 's  of  the  world.  Generally  speaking 

though,  they  have  not  thought  about  it  to  the  extent,  that, 

I  don't  know,  they  Just  use  what  we  have.  The  first  five 
attempts  they  will  try  to  use  exactly  what  we  have.  Then  it 
starts  to  sink  in  because  then  that's  where  we  get  drug  into 

it,  like  we  did  on  the  _ .  We've  worked  with  the  WBS 

here  in  the  Cost  Shop.  We  know  what  it  is,  and  we  know  how 
to  get  rid  of  some  false  summing  levels,  so  we  can  more 
rapidly  make,  help  them  get  a  WBS  that  ACCM  will  buy... 
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Appendix  B:  Expert  System  Know! edge  Base 


The  knowledge  base  created  as  the  result  of  this 
experiment  can  be  found  on  the  following  pages.  VP  Expert 
was  the  expert  system  shell  that  was  used.  The  knowledge 
base  is  printed  in  its  entirety  and  has  been  field  tested  as 
stated  in  the  text  of  this  report.  No  identified 
discrepancies  have  be  corrected  in  this  version. 

Captions  have  been  inserted  within  black  boarders  to 
explain  some  of  the  required  syntax  for  developing  a 
knowledge  base  for  an  expert  system.  The  reader  is  reminded 
that  the  syntax  used  is  specific  to  the  VP  Expert  expert 
system  shel 1 . 
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ENDOFF; 


RUNTIME; 

ACTIONS 
DISPLAY" 

Welcome  to  an  experimental  expert  system  to  aid  the 
development  of  a  draft  work  breakdown  structure.  This 
system  is  limited  to  the  ELECTRONICS  or  MISSILE  categories 
of  defense  materiel  items.  Please  feel  free  to  experiment 
with  this  system,  but,  recognize  that  it  is  limited  in 
capability.  Much  additional  research  is  needed  to  make  this 
system  fully  functional. 

To  answer  the  questions  presented  by  this  expert  system, 
either  input  an  appropriate  name  when  asked,  or  if  the 
question  can  be  answered  by  the  yes  or  no  menu,  use  the 
arrow  keys  and  the  enter  key  to  indicate  your  response.  An 
unknown  answer  (indicated  by  a  ’ ? ’  mark)  to  any  ’Yes  or  No’ 
question  will  be  treated  as  a  ’No’.  That  particular 
element  will  be  left  off  the  final  work  breakdown 
structure. 

Please  strike  any  key  to  continue  the  consultation.'' 

II 

els 

DISPLAY" 

The  expert  system  will  present  you  with  the  work  breakdown 
structure  element  and  the  proper  numbering  of  that  element 
on  the  monitor.  Additionally,  the  expert  system  will  also 
cause  the  printer  to  print  the  element  and  number. 

Please  strike  any  key  to  continue  the  consultation.'' 

I* 

els 


ACTIONS:  Identifies  and  begins 
the  series  of  rules  and  user 
proapts  that  aake  up  the  expert 
systea  consultation.  (13:3.2) 


CLS 

X=0 

Y=0 

FIND  program 

FIND  category 

CLS 

DISPLAY  "Your  system 
defense  materiel  items. 


progr am 
category 

"Your  system  belongs  in  the  (category)  CATEGORY  of 


Press  any  key  to  continue  the  consultation.''" 
CLS 


» 
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FIND 

WBS  type 

FIND 

FIND 

CLS 

PRINTON 

occ 

ogc 

FIND:  Identifies  variables  in  the 
VP-Expert  knowledge  base  that  aust 
be  identified  to  coaplete  a 
consultation.  These  variables  aay 
be  identified  through  user  inputs 
or  froa  the  execution  of  eipert 
systea  rules.  (13:9.43) 

DISPLAY 

"l.CX>  <p 

rogram>  CWBS_type>" 

FIND 

FIND 

FIND 

FIND 

FIND 

f 

pr oduc  t 

1  ev-el  _3 
producta 

1  evel  _3 

1  evel  _2 

ASK:  If  the  expert  systea 
inference  engine  cannot  find  a 
value  for  a  variable  identified  in 
a  FIND  clause,  an  ASK  clause  can 
be  used  to  prcapt  the  expert 
systea  user  for  the  variable 
value.  (13:9.9) 

ASK  program:  "What  is  the  name  of  your  acquisition 
prograiri7"  ; 


ASK  WBS_type:  "Do  you  wish  to  produce  a  draft  work  breakdown 
structure  for  development  <FSD)  or  for  production^ 

"  • 
t 

ASK  occ: " 

Do  you  wish  to  include  other  contract  costs  in  your  WBS"7 

"  « 
f 

ASK  ogc:"Do  you  wish  to  include  other  government  costs  in 
your  WBS"7 


CHOICES:  Used  m  conjunction  with 
in  ASK  stateaent  to  restrict  the 
user’s  input  to  one  or  a 
coabination  of  specific  values  for 
a  given,  variable.  (13:5.23) 


CHOICES  WBS_type:  Development,  Production; 

CHOICES  electronic,  spac e_veh 1 c 1 e,  sensors,  comml, 
autodata,  programs,  displays,  auxequip:  YES, NO; 

CHOICES  guidance,  payload,  occ,  ogc :  YES, NO; 

CHOICES  shroud,  testequip,  trainequip,  auxequip2, 
sur  ve 1 1 1 anc  e , 

1 aunc h_c ont r ol ,  comm2:  YES,  NO; 

CHOICES  PMD,  1  aunch_equip,  auxequip3,  dataprocess:  YES,  NO; 
CHOICES  PMD 1 ,  PMD2,  PMD3,  experience,  subc ont r ac t or s , 
other_sof tware, 

material,  destination:  YES, NO; 


59 


CHOICES  PMD4,  PMD5,  PMD6,  PMD7,  val ue , new_t ech ,  common, 
command:  YES,  NO; 

CHOICES  TP,  ED,  MD,  SD:  YES, NO; 

CHOICES  E,  S,  F:  YES,  NO; 

CHOICES  DTE,  OTE,  mockups,  tesuoport:  YES, NO; 

RULE  10 

IF  el ec t r on  1 c =yes 

THEN  cat  egor y=ELECTRON ICS; 

ASK  electronic : " 

Is  {program}  classified  as  the  electronics  portion  of  a 
defense  materiel  item  AND  is  the  system  unique  or  used  as  a 
building  block  for  several  systems  not  accounted  for  in 
those  systems? 

"  . 

t 


RULE  20 

IF  spac e_veh 1 c 1 e=yes 

THEN  c at egor y=MI SS ILE; 


ASK  space_vehicle: 

Does  {program}  employ  an  unmanned,  self-propelled  air/space 
vehicle  AND  does  this  air/space  vehicle  navigate,  penetrate 
and  produce  a  desired  effect  on  selected  targets0 

•»  • 
t 


RULE:  A  precise  or  senes  of 
preaises  and  a  conclusion.  The 
conclusion  is  true  only  if  the 
preaises  are  satisfied.  The 
conclusion  can  be  used  to  assign 
values  to  variables.  (13:9.85) 


RULE  30 

IF 

category=electronics 

THEN 

pr oduc  t  =PME 

X=(X+1 ) 

FIND 

PME 

DISPLAY 

”  1  . { X  }  {PME}”; 

ASK  PME: 

1 1 

What  is 

the  name  of  your  prime  mission 

••  • 
9 


RULE  40 
IF 

THEN 

FIND 

DISPLAY 


category=missile 
product=a i r _veh  i  c  1  e 
X= ( X+l ) 
air  _veh i c 1 e 
" 1 . { X  }  {  a  i  r _veh i c 1 e }  "  ; 


DISPLAY:  Used  to  display  teit  to 
the  expert  systea  user.  This  text 
aust  be  enclosed  within  the  double 
quotation  aarlts.  (13:9.34) 
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ASK  ai r_veh ic  1  e:  " 

What  do  you  wish  to  name  the  air  vehicle  element  of 
{program}-7 

M  » 
f 


RULE  50 


IF- 

THEN 

DISPLAY 

RESET 


c ategory=el ec t ron i c s  AND 
sensor s=yes 
1 evel  _3=Sensors 
Y= (Y+l ) 

"1.<X>.<Y>  {  1  evel  _3 > " 

1 evel  _3 ; 


ASK  sensors: " 

Does  <PME>  have  equipment  to  detect  and  indicate  terrain 
features,  the  presence  of  military  targets,  or  other  natural 
or  man-made  objects  be  means  of  energy  emitted  or  reflected 
by  those  objects? 

"  • 

7 


RULE  60 


IF 

THEN 

DISPLAY 

RESET 


category=electronics  AND 
c  omm 1 =yes 

1 evel _3=Commun i c  at i ons 
Y= (Y+l ) 

"1. {  X  } . CY>  { 1 evel  _3  } " 

1  evel  _3 ; 


ASK  c  omm 1 :  " 

Does  CPME}  have  the  means  to  receive  or  transmit  messages  of 
data  from  one  person  or  place  to  another-7 


RULE  70 


IF 

THEN 

DISPLAY 

RESET 


category=el ectron ics  AND 
aut  odat  a=yes 

level  _3=Aut  omat i c  _Dat  a_Proc ess l ng 
Y= ( Y+l ) 

”1.<X}.CY>  { 1 evel  _3 > " 

1  evel  _3 ; 


ASK  aut  odat  a :  " 

Does  <PME>  have  equipment  consisting  of  input,  storage, 
computing,  control,  and  output  devices  which  use  electronic 
circuitry  to  aut omat i c a  1 1 y  perform  arithmetic  and/or  logical 
operations  by  means  of  internally  or  externally  controlled 
programmed  instructions7 
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RULE  80 


IF  category=electronics  AND 

WBS_type=devel opment  AND 
pr ogr  ams=yes 

THEN  1  evel  _3=Computer _Progr  ams 

Y= ( Y+ 1 ) 

DISPLAY  "  l.(X>.CY>  C 1 evel _3  > " 

RESET  1 evel _3; 

ASK  programs: " 

Does  CPMEXprogramJ  contain  programs  and  routines  consisting 

of  a  deck  of  punched  cards,  magnetic  or  paper  tapes, 

read-only  memory  units,  or  other  physical  medium  containing 

a  sequence  of  instructions  and  data  in  a  form  suitable  for 

insertion  into  a  computer  and  used  to  direct  the  computer 

to  perform  a  desired  operation  or  sequence  of  operations? 

»»  • 
t 


RULE  90 

IF  category=el ectronics  AND 

d i sp 1 ays=yes 

THEN  1 evel _3=Data_Displ ays 

Y=(Y+1 ) 

DISPLAY  " l.CX}.{Y)  { 1 evel _3> " 

RESET  1 evel _3; 

ASK  d ispl ays: " 

Does  f  PME  >  contain  a  visual  presentation  of  processed  data 
by  specially  designed  electronic  or  el ec t romechan i c al 
devices  through  interconnections  with  digital  computers  or 
component  equipments  NOT  to  include  line  printers  or  punched 
cards7 


RULE  100 

IF  c at egor y=el ec t r on i c s  AND 

auxequip=yes 

THEN  1 evel _3=Aux i  1  i ary_Equ ipment 

Y=(Y+1 ) 

DISPLAY  "l.CX>.<Y>  < 1 evel _3> " 

RESET  1 evel _3; 

ASK  auxequip: " 

Does  {Pf*lE>  contain  common  or  multi-usage  equipments  used  to 
augment  the  functional  performance  of  several  level  3 
elements  and  which  are  NOT  homogenous  to  the  designated 

level  3  elements'7 

"  . 

9 
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RULE  110 


IF  category=MISSILE  AND 

produc  t  =a i r _veh ic  1  e 
THEN  FIND  stages; 

ASK  stages: " 

How  many  separable  propulsion  stages  does  ( a i r _veh i c  1  e> 

have?  (not  to  exceed  4) 

"  • 
t 


RULE  120 
IF 

THEN 

DISPLAY 

RESET 

RULE  130 

IF 

THEN 

DISPLAY 

RESET 

RULE  140 

IF 

THEN 

DISPLAY 

RESET 

RULE  150 

IF 

THEN 

DISPLAY 

RESET 

RULE  160 

IF 

THEN 


category=mi ssi 1 e  AND 
st  ages= 1 

1  evel  _3=Pr opul s i on 
Y=(Y+1 ) 

"1.{X}.(Y>  {  1  evel  _3 > " 

1  evel  _3; 


category=missi 1 e  AND 
stages  >  1 
1 evel  _3=Stage_l 
Y=  ( Y  + 1  ) 

" l.CXJ.CYJ  {  1  evel  _3> " 

1  evel  _3; 


c ategory=mi ss i 1 e  AND 
stages  >=  2 
1 evel  _3=St  age_2 
Y= (Y+l ) 

"1.<X>.(Y>  { 1 evel  _3  >  " 

1  evel _3; 


c ategory=mi ss i 1 e  AND 
stages  >=  3 
1  evel  _3=St age_3 
Y= (Y+l ) 

"l.tX>.CY>  i 1 evel  _3  > " 

1  evel _3; 


c ategory=m l ss l 1 e  AND 
stages  >=4 
1 evel  _3=St  age_4 
Y=(Y+1 ) 
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DISPLAY  "l.CX>.CY>  <level_3>" 

RESET  level  _3; 


RULE  170 
IF 


THEN 

DISPLAY 
RESET 

ASK  programs:  " 

Does  {program}  contain  programs  and  routines  consisting  of  a 
deck  of  punched  cards,  magnetic  or  paper  tapes,  read-only 
memory  units,  or  other  physical  medium  containing  a  sequence 
of  instructions  and  data  in  a  form  suitable  for  insertion 
into  a  computer  and  used  to  direct  the  computer  to  perform  a 

desired  operation  or  sequence  of  operations? 

"  . 

t 

RULE  180 

IF  category=missi 1 e  AND 

guidance=yes 

THEN  1  evel  _3=Guidance_and_Control _Equipment 

Y= ( Y+ 1 ) 

DISPLAY  " 1 . <X>. {Y>  { 1 evel  _3> " 

RESET  1 evel _3 ; 

ASK  gui dance: " 

Does  { a l r _veh i c  1  e }  have  the  means  for  generating  or 
receiving  guidance  intelligence,  conditioning  that 
intelligence  to  produce  signals  and  generating  appropriate 

control  forces'7 

"  • 

9 


RULE  190 
IF 

THEN 

DISPLAY 
RESET 

ASK  payl oad :  " 

Does  < a l r _veh i c 1 e >  have  a  means  to  produce  a  destructive 

effect  on  the  target  at  the  terminal  point  of  flight7 
1 1  fl 
* 


category=missi 1 e  AND 
payl oad=yes 

1  evel  _3=Launched_Payload 
Y=(Y+1  ) 

” 1.<X>.{Y>  {  1  evel  _3 > " 

level  3; 


category=MISSILE  AND 
WBS_t ype=devel opment  AND 
pr ogr ams=yes 

1  evel _3=Computer_Programs 
Y= ( Y+ 1 ) 

"1.{X>.CY>  {  1  evel  _3> " 

level  3; 
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RULE  200 


category=missi 1 e  AND 
shroud=yes 

1  evel  _3=Payl oad_Shr oud 
Y= ( Y+l ) 

" l.CX>.<Y>  < 1 evel  _3> " 

1  evel  _3; 

ASK  shroud : " 

Does  Cair_vehicle)  have  a  protective  enclosure  for 
safeguarding  its  payload  during  the  severe  environments  of 
launch  and  flight? 

»»  . 

f 


IF 

THEN 

DISPLAY 

RESET 


RULE  210 


IF 

THEN 

DISPLAY 

RESET 


category=missi 1 e  AND 
testequip=yes 

1  evel  _3=A irborne_T est_Equ i pment 
Y=(Y+1) 

" 1 . (  X  > . CY>  <  1  evel  _3> " 

1  evel  _3; 


ASK  testequip :  ” 

Does  < air_veh i c 1 e)  have  an  exercise  warhead  that  is  suitable 
for  developmental  firing? 

■i . 

r 


RULE  220 


IF 

THEN 

DISPLAY 

RESET 


c at egor y=mi ss l 1 e  AND 
trainequip=yes 

1  evel  _3=Ai rbor ne_Tr a i n i ng_Equi pment 
Y=(Y+1) 

Ml.fX>.<Y>  < 1 evel  _3  > " 

1 evel  _3; 


ASK  trainequip : " 

Does  (  a  i  r_veh  i  c  1  e>  have  an  exercise  warhead  that  is  suitable 

for  training? 

"  . 

f 


RULE  230 


IF 

THEN 

DISPLAY 


c at egor y=mi ss i 1 e  AND 
auxequ ip2=yes 

1 evel _3=Aux i 1 i ar y_Equ i pment 
Y= ( Y+l ) 

"l.(X).{Y>  (level  3>" 
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RESET  level _3; 

ASK  auxequip2: " 

Does  { a i r _veh ic  1  e>  have  additional  equipment,  that  is 
excluded  from  the  other  level  3  elements,  but  that 
supplements  or  provides  service  to  these  other  elements 
WITHIN  the  air  vehicle,  (air_vehicle>? 

"  . 

f 


RULE  240 

IF  c at egor y=miss i 1 e  AND 

c  ommand=yes 

THEN  product  a=Command_and_Launc  h_Equ l pment 

X=(X+1) 

DISPLAY  "1.<X>  (producta)" 

RESET  Y; 

ASK  c  ommand : " 

Does  (program)  have  subsystems  installed  at  a  launch  site  or 
aboard  launch  vehicles  for  mobile  systems,  required  to 
store,  make  ready,  and  launch  (air_vehicle>,  the  air  vehicle 
of  this  missile  system? 

» . 

f 


RULE  250 

IF  c at egor y=m i ss i 1 e  AND 

pr oduc  t a=Command_and_Launc  h_Equ i pmen t  AND 
survei 1 1 ance=yes 

THEN  1 evel _3=Survei 1 1 anc e_and_Tr ac  k ing_Sensors 

Y= (Y+l ) 

DISPLAY  " 1. {  X  ) . CY>  {  1  evel  _3 > " 

RESET  1  evel  _3 ; 

ASK  surveillance: " 

Does  {program)  have  sensors  to  support  defensive  missile 
systems  by  maintaining  surveillance  against  incoming  targets 
and  providing  the  data  required  for  targeting,  launch, 
midcourse  guidance,  and  homing  where  such  capability  IS  NOT 
self-contained  aboard  the  missile  system,  { a i r_veh l c  1  e)? 

li  . 

t 


RULE  260 

IF  c ategory=mi ss l 1 e  AND 

product  a=Command_and_Launch_Equi pment  AND 
1  aunch_control=yes 

THEN  1  evel _3=Launc h_and_Gu l danc e_Equ i pment 

Y=CY+1 ) 
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DISPLAY  " l.CX>.CY>  <level_3>" 

RESET  level  _3; 

ASK  1 aunch_control : " 

Does  {program)  have  the  means  to  enable  targeting  of  missile 
air  vehicles,  to  enable  launch  decisions  to  be  made,  and  to 
enable  command  to  launch? 


RULE  270 

IF  category=mi ssi 1 e  AND 

produc t a=Command_and_Launc h_Equipment  AND 
comm2=yes 

THEN  1  evel _3=Commun i c  at l ons 

Y=(Y+1 ) 

DISPLAY  "  l.CX).<Y>  {  1  evel  _3  >  11 

RESET  1  evel  _3 ; 

ASK  comm2: " 

Does  {program}  have  the  means  to  distribute  intelligence 
within  the  missile  system? 

>• . 

f 

RULE  280 

IF  c ategor y=missi 1 e  AND 

product  a=Command_and_Launch_Equipment  AND 
dat  apr oc  ess=yes 

THEN  1 evel  _3=Dat  a_Pr oc  ess i ng_Equ i pment 

Y=(Y+1 ) 

DISPLAY  "1.{X>.{Y>  { 1 evel  _3  > ” 

RESET  1 evel  _3; 

ASK  dataprocess: " 

Does  {program}  have  the  means  to  condition  data  generated  at 
the  launch  site  or  aboard  the  launch  vehicle  of  a  system 
employing  mobile  launch  capability,  or  data  received  from 
associated  systems  to  accommodate  the  needs  of  launch  and 

guidance  control? 

"  • 
t 

RULE  290 

IF  c at egor y=m i ss i 1 e  AND 

produc t a=Command_and_Launc h_Equ i pment  AND 
1  aunc  h_equ i p=yes 

THEN  1 evel  _3=Launc h_Equ i pment 

Y= < Y+ 1 ) 

DISPLAY  " l.(X}.{Y}  {  1  evel  _3 } ” 

RESET  1  evel  _3; 
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ASK  1 aunc h_equ i p :  " 

Does  (program)  have  the  means  to  launch  the  missile  from 

stationary  or  mobile  launch  platforms? 

. 

i 


RULE  300 


IF 

THEN 

DISPLAY 

RESET 


category=miss i 1 e  AND 

produc  t  a=Command_and_Launch_Equipment  AND 
auxequip3=yes 

1  evel _3=Aux i  1  iary_Equipment 
Y= ( Y+l ) 

" 1 .  {  X  } . {  Y  }  <  level  _3>" 

1  evel  _3; 


ASK  auxequip3: " 

Does  {program}  have  general -pur pose  ground  equipment  used  to 
supplement  the  various  operational  equipments  of  the 
{products)0 

•• . 

f 


RULE  310 


IF 

THEN 

DISPLAY 

RESET 


category=electronics  OR 
category=missile 
1  evel  _3=Hardware_Integrat i on 
Y=  ( Y+l ) 

" 1 . {X). {Y>  {  1  evel _3> " 

1  evel _3; 


RULE  320 


IF 


THEN 

DISPLAY 

RESET 


category=el ectronics  OR 
category=missi 1 e  AND 
WBS_t ype=devel opment  AND 
programs=yes 

1 evel _3=Sof tware_Integrat ion 
Y= (Y+l ) 

" 1.{X).{Y)  { 1 evel  _3> " 

1  evel  _3; 


RULE  330 

IF  c at egory=el ec t r on l c s  OR 

c at egor y=mi ss i 1 e  AND 
WBS_t ype=devel opment  AND 
pr ogr  ams=yes 
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THEN  1 evel _3=Hardwar e_So f t war e_Int egr at  ion 

Y=CY+1) 

DISPLAY  " 1. <  X  > . CY>  C 1  evel _3> ” 

RESET  1  evel  _3; 


RULE  340 


IF 

THEN 

DISPLAY 

RESET 


WBS_type=devel opment  AND 
new_tech=yes 

1  evel  _2=Precontract_Award_Studies 
X=(X+1) 

" l.(X>  i 1 evel  _2> " 

1  evel  _2; 


ASK  new_tech:  " 

Does  (program)  contain  radical,  state  of  the  art, 
technology  or  does  (program)  represent  a  signific 
up-grade  to  existing  technology? 

■I . 
r 


RULE  350 

IF 

WBS_t ype=devel opment  AND 

THEN 

oc  c  =yes 

1  evel  _2=Sys_Engineer ing_and_Proj 

DISPLAY 

X=(X+1 ) 

M 1 . €  X  >  (  1  evel  _2  > " 

RESET 

1  evel  _2; 

RULE  360 

IF 

WBS_t ype=devel opment  OF: 

THEN 

WBS_type=pr oduc t i on  AND 
oc  c  =yes 

1  evel  _2=Dat  a 

DISPLAY 

X=(X  +  1  ) 

"l.CX>  ( 1 evel  _2  >  " 

RESET 

Y 

FIND 

1  evel  _3d 

RESET 

1 evel  _2; 

RULE  370 

IF 

level_2=data  AND 

THEN 

TP=yes 

1  evel _3d=Techn i c al _Pub 1 ications 

DISPLAY 

Y=  (  Y+ 1  ) 

"1.<X>.(Y>  (  1  evel  _3d  > " 

RESET 

1 evel  _3d  j 

new 

ant 
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ASK  TP: " 

Does  (program)  require  the  development  of  formal  technical 
order s/manual s  for  the  installation,  operation,  maintenance, 
overhaul,  training  and  reference  of  hardware  and  computer 
programs? 

»»  * 

t 


RULE  380 
IF 

THEN 

DISPLAY 
RESET 

ASK  ED: " 

Does  the  Program  Office  want  engineering  drawings, 
associated  lists,  specifications,  and  other  similar 
documentat ion? 

"  . 

f 


RULE  390 
IF 

THEN 

DISPLAY 
RESET 

ASK  MD: " 

Does  the  Program  Office  want  those  data  items  necessary  for 
con f igurat ion  management,  cost,  schedule,  contractual  data 
management ,  and  program  management7 


RULE  400 
IF 

THEN 

DISPLAY 
RESET 

ASK  SD:  " 

Does  the  Program  Office  want  those  data  items  to  document 
the  logistics  support  planning  and  provisioning  process7  " 


1  evel _2=dat  a  AND 
SD=yes 

1  evel  _3d=Support  _Dat  a 
Y= ( Y  + 1  ) 

"  l.CXJ.CY}  {  level _3d > ” 

level  3d; 


level_2=data  AND 
MD=yes 

1 evel _3d=Management  _Dat  a 
Y= ( Y+ 1 ) 

” l.(X>.<Y>  ( 1 evel  _3d  > " 

level  3d; 


level_2=data  AND 
ED=yes 

1 evel _3d=Eng ineer ing_Dat  a 
Y=(Y+1) 

"1.<X>.<Y>  ( 1 evel  _3d  > ” 

level  3d; 
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RULE  410 


IF  occ=yes  AND 

PMD2=yes 

THEN  1  evel  _2=Data_Rights 

X=(X+1 ) 

DISPLAY  "1.<X>  { 1  evel _2  > " 

RESET  1  evel  _2; 

ASK  PMD2 : " 

Does  the  PMD  or  the  System  Program  Office  indicate  that  data 

rights  are  required? 

••  • 
t 


RULE  420 
IF 


T!  iEN 

DISPLAY 

RESET 

FIND 

RESET 


RULE  430 
IF 

THEN 

DISPLAY 
RESET 

ASK  E:  " 

Are  distinctive  items  of  training  equipment  needed  to  meet 

specific  training  objectives? 

"  . 

F 


RULE  440 

IF  1 evel  _2=Tra in ing  AND 

S=yes 

THEN  1 evel _3t =Ser v i c es 

Y= ( Y+l ) 

DISPLAY  ” 1 . CX>. CY>  {  1  evel  _3t > " 

RESET  1  evel  _3t ; 


1  evel _2=Tr a l n i ng  AND 
E=yes 

1  evel  _3t  =Equ l pment 
Y=(Y+1 ) 

"1.<X>.CY>  (  1  evel  _3t  > " 

1 evel  3t ; 


WBS_type=devel opment  OR 
WBS_t ype=pr oduc t i on  AND 
oc  c -yes 

1  evel _2=Tr a i n i ng 
X=(X  +  i  ) 

"1.{X>  (  1  evel  _2 > " 

Y 

1  evel  _3t 
level  2; 


Are  services,  devices,  accessories,  and  aids  necessary  to 

accomplish  training  objectives? 

•  *  • 

F 


RULE  450 
IF 

THEN 

DISPLAY 
RESET 

ASK  F:  " 

Are  special  constructions  necessary  to  accomplish  training 
objec t  x ves? 

(specifically  brick  and  mortar  type  construction) 

11 . 
t 


RULE  4£>0 
IF 

THEN 

DISPLAY 

RESET 

FIND 

RESET 


RULE  470 
IF 

THEN 

DISPLAY 
RESET 

ASK  DTE: 1 

Does  (program)  require  tests  to: 

a)  Demonstrate  that  design  risks  have  been  minimized? 

b)  Demonstrate  that  the  engineering  design  and  development 
process  is  complete? 

c)  Demonstrate  that  the  system  will  meet  specifications? 

d)  Estimate  the  (pme > ( a i r _veh i c 1 e ) f s  military  utility? 

e)  Provide  test  data  with  which  to  examine  and  evaluate  the 
tradeoffs  against  specification  requirements,  life  cycle 
cost,  and  schedule’’ 


1  evel  _2=System_T est_and_Eval uat ion  AND 
DTE=yes 

1  evel _3te=Devel opment _Test _and_Eval 
Y=  ( Y+l  ) 

" l.(X>.(Y>  (  1  evel  _3t  e  > " 

level  3te; 


WBS_t ype=devel opment  AND 
o»_c  =yes 

1  evel  _2=System_Test_and_Eval uat ion 
X=CX+1 ) 

" l.CX>  ( 1 evel  _2  ) " 

Y 

1 evel  _3te 
level  2; 


1  evel  _2=Tr a  in i ng  AND 
F =yes 

1  evel  _3t=Fac i  1  it ies 
Y=(Y+1 ) 

"1. (X>. (Y>  ( 1 evel  _3t  > " 

level  3t ; 
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RULE  480 


IF 

THEN 

DISPLAY 

RESET 


1  evel  _2=System_Test_and_Eval uat ion  AND 
0TE=yes 

1  evel  _3te=0perat ional  _Test _and_Eval 
Y=  ( Y+l ) 

"1.<X>.<Y>  <  1  evel  _3t e  > " 

1  evel  _3te; 


ASK  OTE: " 

Are  tests  to  be  conducted  by  agencies  other  than  the 
developing  command  to  assess  (programl’s  military  utility, 
operational  effectiveness,  operational  suitability, 
logistics  supportabi 1 ity,  cost  of  ownership,  and 
need  for  any  modifications? 

It  , 

t 


RULE  490 


IF 

THEN 

DISPLAY 

RESET 


1 evel _2=System_Test_and_Eval uat ion  AND 
moc  kups=yes 
1 evel _3t  e=Moc kups 
Y=(Y+1) 

"1.<X>.<Y>  {  1  evel  _3te  > " 

1  evel  _3te; 


ASK  moc  kups : " 

Is  there  a  need  to  design  and  produce  system  or  subsystem 
moc kups  which  have  special  contractual  or  engineering 
significance,  or  which  are  NOT  required  solely  for  the 

conduct  of  one  of  the  above  elements  of  testing? 

"  • 
t 


RULE  500 


IF 

THEN 

DISPLAY 

RESET 


1  evel  _2=Sy5tem_Test_and_Eval uat ion  AND 
tesupport=yes 

1 evel _3t  e=Test_and_Evaluati on_Suppor t 
Y=  ( Y+l  ) 

”1. <  X  > . CY>  ( 1 evel _3t  e  > " 

1  evel  _3te; 


ASK  tesupport : " 

Will  contractor  support  be  necessary  to  operate  and  maintain 
systems  during  testing  and  evaluation  which  are  NOT  consumed 
during  a  particular  category  of  testing?  Operator  and 
maintenance  personnel,  consumables,  special  fixtures, 
special  instrumentation,  etc.  which  are  used  in  a  single 
element  of  testing  (such  as  DT&E)  should  not  be  included 
under  this  element. 

•'  • 
r 
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RULE  510 


IF  WBS_t ype=pr oduc t 1  on  AND 

oc  c  =yes 

THEN  1  evel _2=Sust a i n 1 ng _Eng_and_Pr o j  Nqt 

X=(X+1) 

DISPLAY  "l.CX)  C 1  evel _2  > " 

RESET  1 evel _2 ; 


RULE  520 
IF 

THEN 

DISPLAY 

RESET 


RULE  530 

IF  occ=yes  AND 

PND=yes 

THEN  level_2=Aircraft_Integrat ion 

X=(X+1) 

DISPLAY  "l.(X)  ( 1  evel  _2> " 

RESET  1  evel _2; 


ASK  PND: " 

Does  the  PND  or  the  Systen.  Program  Office  indicate  that 
aircraft  integration  is  required7 


RULE  540 

IF  occ=yes  AND 

PND 1 =yes 

THEN  leveI_2  =  Per.  j]  i  ar_Support_Equipment 

X=(X+1 ) 

DISPLAY  "1.<X>  (  1  evel  _2  > " 

RESET  Y 

FIND  1  evel _3PSE 

RESET  1  evel  _2; 

ASK  PND  1 :  " 

Does  the  PND  or  the  System  Program  Office  indicate  that 
peculiar  support  equipment  is  r->ouired  or  is  unique 
equipment  required  to  support  and  maintain  {program)  and 
which  has  an  application  unique  to  (program)7  *’ ; 


WBS_t ype=pr oduc t i on  AND 
oc c =ye s 

1 evel _2=Ac  c  ept  anc  e_T  est i ng 
X=(X+1) 

"l.(X)  (  1  evel  _2  > " 

level  2: 
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RULE  550 


01=organizat lonal  DR 
0 1 =bot  h 

1 evel _3PSE=0rgan izat lonal  _or_Intermed 2  at  e 
Y=(Y+1) 

“ 1.<X>.CY>  {  1  evel  _3PSE  >  " 

1  evel  _3PSE; 


RULE  560 
IF 

THEN 

DISPLAY 
RESET 

ASK  01 : " 

Is  the  Peculiar  Support  Equipment  required  at  the 
organ i z at ional / int ermed l at e,  depot  level  or  both? 

"  . 

r 

CHOICES  01:  Organizational ,  Depot,  Both; 


RULE  570 

IF  ogc=yes  AND 

c  ommon=yes 

THEN  level _2  =  Common_Suppor t  _Equ l pment 

X  =  C  X  +  l ) 

DISPLAY  "l.(X}  {  1  evel  _2  > " 

RESET  Y 

FIND  1 evel  _3c 

RESET  1  evel  _2; 

ASK  c  ommon :  " 

Does  {program}  require  additional  support  equipment  that  can 
be  identified  through  national  stock  number  listings? 

(input  from  a  logistician  recommended} 

"  . 
f 


RULE  5R0 


0 I c =or gan i z at l onal  OR 
0 1 c  =bot  h 

1  evel  _3c  =0rganizat ional  _or _I nt ermed l at e 
Y= ( Y+  1  ) 

*'  1 .  CX}.  <Y>  <  1  evel  _3c  }  " 

level  _3c  ; 


IF 

THEN 

DISPLAY 

RESET 


0I=Depot  OF 
0I=both 

level  3PSE=Depot 
Y=(Y+T) 

"l.CXl.CY}  {  1  evel _3PSE } " 

1 evel  _3PSE ; 


IF 

THEN 

DISPLAY 

RESET 
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RULE  590 


IF 

THEN 

DISPLAY 

RESET 


0Ic=Depot  OR 
0 1 c  =bot  h 
1  evel _3c  =Depot 
Y— ( Y+l  ) 

'*  1  .  CX>.  (Y} 

1  evel  _3c ; 


(level  3c  > " 


ASK  01 c: " 

Is  the  Common  Support  Equipment  required  at  the 

organ i z at i onal / int er med l at e ,  depot  level  or  both7 

11  • 

f 

CHOICES  01c:  Or gan i 2 at i onal ,  Depot,  Both; 


RULE  600 


IF 

THEN 

DISPLAY 

RESET 


occ=yes  AND 
exper l enc  e  =  no 
1  evel  _2=T ype_l _Tr a l n i ng 
X=CX+1 ) 

" 1 . { X  }  ( level  _2>" 

1  evel  _2; 


ASK  experience:" 

Does  *-he  Air  Force  have  experience  with  this  type  of  weapon 

syst  em7 

»»  . 

f 


RULE  610 


IF 

THEN 

DISPLAY 

RESET 


occ=yes  AND 
subcontractors=yes 
1  evel  _2=Wor k_Measur ement 
X= ( X+l ) 

" l.(X>  (  1  evel  _2 ) " 

1  evel  _2 ; 


ASk  subcontractors: " 

Will  the  prime  contractor  employ  subcontractors  to  help 
fulfill  the  requirements  for  (program}7 


RULE  620 

IF  occ=yes  AND 

WBS_t ype=devel  opment  AND 
programs=yes 
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THEN 


1  evel  _2=Sof tware_Integrat  ion_Lab 
X=(X+1) 

DISPLAY  "1.{X>  {  1  evel _2> “ 

RESET  1 evel _2; 

RULE  630 

IF  WBS_t ype=devel opment  AND 

occ=yes  AND 
programs=no  AND 
other  __so  ft  war  e=yes 

THEN  1  evel _2=Sof t ware_Integr at ion_Lab 

X=(X+1) 

DISPLAY  "1.<X>  {level  _2> ” 

RESET  1 evel _2 ; 

ASK  other_sof t ware: " 

Does  {program}  require  software  development  for  software 

that  will  be  critical  for  system  flight  operations? 


RULE  640 
IF 


THEN 

DISPLAY 

RESET 


RULE  650 

IF  occ=yes  AND 

WBS_t ype=pr oduc t l on  AND 
mater ial =yes 

THEN  1  evel  _2= Inter i m_Cont r ac  t or _Suppor t 

X=(X+l ) 

DISPLAY  "1.{X>  { 1  evel _2  > " 

RESET  1 evel _2; 

ASK  mater ial s " 

Will  the  c ont r ac t or ( s )  furnish  material  and  maintenance 

pending  assumption  of  these  services  by  the  military? 

*»  • 

9 


RULE  660 

IF  occ=yes  AND 


occ=yes  AND 

WBS_type=devel  opment  AND 
progr ams=yes  OR 
other__sof  t  ware=yes 
1  evel  _2=Sof t ware_Support 
X=(X+1 ) 

"1.{X>  { 1 evel  _2  >  " 

level  2; 
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THEN 


WBS_type=product ion  AND 
dest inat ion=yes 

level  _2=F irst_Dest inat ion_Transpor tat  ion 
X=(X+1) 

DISPLAY  "l.CX>  ( 1  evel  _2> " 

RESET  level_2; 

ASK  destination:" 

Will  (program)  be  delivered  from  a  procurement  source 
outside  the  Department  of  Defense  inventory  to  the  first 
point  of  use  or  storage  for  subsequent  distribution  within 
the  Air  Force ° 

»»  • 

f 


RULE  670 

IF 

occ=yes  AND 

WBS_t ype=pr oduc t ion 

THEN 

1  evel _2=War r  ant  y 
X=(X+1) 

DISPLAY 

"l.CX>  {level 

_2  >  " 

RESET 

1  evel  _2; 

RULE  680 

IF 

occ=yes  AND 

WBS_t ype=pr oduc t ion 

THEN 

level  2=Hardware  Qualitv  Audit 

X=(X  +  1  ) 

DISPLAY 

"1.{X>  {level 

_2  >  " 

RESET 

1  evel  _2; 

RULE  690 

IF 

ogc=yes  AND 

WBS_t  ype=devel opment 

THEN 

level_2=Fl  ight_Test 
X=(X  +  1  ) 

DISPLAY 

"1.{X>  {level 

_2  >  " 

RESET 

1  evel  _2j 

RULE  700 

IF 

ogc=yes  AND 

WBS_t  ype=devel opment 
WBS_t ype=produc t ion 

OR 

THEN 

1  evel  _2=Award_Fee 

X=( X  +  l  ) 

DISPLAY 

" 1  - { X  >  {level 

_2>" 
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RESET  level _2; 

RULE  710 
IF 

THEN 

DISPLAY 
RESET 

RULE  720 
IF 

THEN 

DISPLAY 
RESET 

RULE  730 
IF 

THEN 

DISPLAY 
RESET 

RULE  740 
IF 

THEN 

DISPLAY 
RESET 

ASK  PMD4:  ” 

Does  the  PMD  or  the  System  Program  Office  indicate  that 

Government  Furnished  Equipment  is  required7 

'•  • 

9 


ogc=yes  AND 
PMD4=yes 

1  evel  _2=Gover nmen t  _Fur n i shed_Equ i pment 
X=(X+1) 

"l.(X>  {  1  evel  _2  >  " 

level  2; 


ogc=yes  AND 

WBS_type=devel opment  OR 
WBS_t  ype=pr oduc  t l on 
1 evel _2=Engineer i ng_Change_Or der s 
X=(X+1) 

" l.(X>  C 1 evel  _2  > “ 

level  2; 


ogc=yes  AND 

WBS_t ype=devel opment  OR 
WBS_type=pr oduc  t ion 
level  2=Mi ssion_Support 
X=(X+1 ) 

"1.{X>  C  1  evel  _2 } " 

1  evel  _2; 


ogc=yes  AND 

WBS_t ype=devel  opment  OR 
WBS_type=product ion 

1  evel _2= Industrial  _Mod_and_I mpr ove_Pr og 
X=(X+1) 

"1.{X>  < 1 evel  _2  >  " 

level  2; 
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RULE  750 


IF 

THEN 

DISPLAY 

RESET 


ASK  PMD5 : " 

Does  the  PMD  or  the  System  Program  Office  indicate  that 
aircrew  training  devices  (simulators)  are  required’ 

ii . 

» 


ogc=yes  AND 
PMD5=yes 

1  evel  _2=Ai  r  c  rew_T  ram  i  ng_Devi  c  es 
X=(X+1) 

" l.(X)  <  1  evel  _2  > " 

1  evel _2; 


RULE  760 


IF 

THEN 

DISPLAY 

RESET 


ogc=yes  AND 
PMD6=yes 

1  evel  _2=Ma l n t  en anc e_Tr  a i n i ng_Devices 
X=  (  X  +  l ) 

" l.(X>  (  level _2>" 

1  evel  _2; 


ASK  PMD6:  " 

Does  the  PMD  or  the  System  Program  Office  indicate  that 
maintenance  training  devices  are  required'’ 

•  i . 

f 


RULE  770 


IF 

THEN 

DISPLAY 

RESET 


WBS_t ype=devel  opment  AND 

ogc=yes  AND 

PMD7=yes 

level _2=0t  her  _t  est  s 
X=(X+1 ) 

”1.(X>  < level  _2>" 

1 evel  _2 ; 


ASK  PMD7: " 

Does  the  PMD  or  the  System  Program  Office  indicate  that 
other  tests,  that  would  not  qualify  as  flight  tests,  are 
required'’ 

>• . 

» 


RULE  780 

IF  ogc=yes  AND 

val ue=yes 

THEN  level  _2=Mi 1 _St andar d_Requ i s i t ion_Program 
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DISPLAY 

RESET 


(level  2  > " 


X=(X+1 ) 

" 1.<X) 

1 evel _2; 

ASK  val ue: " 

Does  the  contractor  require  low  value  government  supply 

1 1  ems? 

•»  • 
t 


RULE  790 
IF 

THEN 

DISPLAY 

RESET 


WBS_type=product ion 
1  evel  _2=Init ial_Spares 
X=(X+1) 

" l.(X>  ( 1 evel  _2  > " 

level  2: 
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Appendix  C:  Concept  Map  of  Expert  * s 
Knowl edge  of  the  Work  Breakdown  Structure 

The  following  pages  are  the  concept  map  that  resulted 
from  the  three  knowledge  acquisition  session  with  one 
expert.  Due  to  the  size  and  complexity  of  the  map  it  cannot 
be  reproduced  on  one  page.  To  reconstruct  the  map,  one  must 
make  copies  of  each  page.  Numbers  on  the  end  of  each  line 
segment  correspond  to  numbers  on  1 ine  segments  on  the 
following  pages.  By  matching  these  numbers,  the  concept  map 
can  be  reconstructed  in  its  entirety. 
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expert  knowledge  to  encode  an  expert  system  rule,  and  the  contribution  each 
technique  made  to  a  final  expert  system  that  represented  an  aggregate  of  all  the 
expert  knowledge  collected.  While  no  one  technique  was  clearly  superior,  used 
together,  concept  mapping  and  interruption  analysis  were  found  to  be  powerful  tools. 
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